Video stream sending method and device

Abstract

Embodiments of the present invention provide a method and apparatus for transmitting a video stream, comprising: obtaining attribute information of each of a plurality of terminals; re-encoding an input video stream based on the attribute information of each terminal to obtain a re-encoded video stream; and transmitting the input video stream or the re-encoded video stream to some or all of the plurality of terminals based on the attribute information of each terminal. This invention solves the prior art problem of poor video stream quality received by a terminal due to compatibility issues between video stream bitstreams. This method achieves the goal of ensuring compatibility between video bitstreams of old and new terminals while ensuring that new terminals can receive a video stream with unchanged image quality even after a delay.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method and device for sending a video stream. Background Art

[0002] The development of information technology has made video conferencing increasingly common in our daily lives and work. In video conferencing applications, to reduce the overall cost of building digital systems, platform-side equipment is often upgraded while client-side equipment remains unchanged. This results in new video conferencing systems containing a variety of conferencing terminals of different brands, models, and generations. This equipment upgrade strategy creates a strong demand for reusing legacy client-side equipment.

[0003] However, building a new video conferencing system that is compatible with a variety of new and existing terminals presents a difficult technical challenge, particularly regarding video stream compatibility. Video streams from terminals from certain manufacturers may not be received and played by all terminals. By re-encoding the video stream, all participating terminals can receive and play the video stream.

[0004] Currently, the input video stream is re-encoded and sent to all receiving terminals participating in the video conference. However, some of these receiving terminals may be capable of receiving the original input video stream. Because the re-encoded video stream has lower quality (e.g., lower resolution), the user experience is degraded for terminals capable of receiving the original input video stream, and there is also the problem of wasted resources.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] The embodiments of the present invention provide a method and apparatus for sending a video stream, so as to at least solve the problem in the related art that the quality of the video stream received by the terminal is poor due to the compatibility problem of the video stream code stream.

[0007] According to one embodiment of the present invention, a method for sending a video stream is provided, comprising: obtaining attribute information of each of a plurality of terminals, wherein the attribute information comprises first attribute information and second attribute information, the first attribute information being used to indicate whether the video stream sent by the terminal needs to be re-encoded, and the second attribute information being used to indicate whether the video stream received by the terminal is a re-encoded video stream, the plurality of terminals participating in the same video conference; re-encoding the input video stream according to the attribute information of each of the terminals to obtain a re-encoded video stream; and sending the input video stream or the re-encoded video stream to some or all of the plurality of terminals according to the attribute information of each of the terminals.

[0008] In an exemplary embodiment, the input video stream is re-encoded according to the attribute information of each terminal to obtain a re-encoded video stream, including: when the input video stream is a video stream sent by a first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, the input video stream is re-encoded to obtain the re-encoded video stream; when the input video stream is a cascaded downstream video stream or a presentation stream, and the second attribute information of a second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded to obtain the re-encoded video stream.

[0009] In an exemplary embodiment, when the input video stream is a video stream sent by a first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, the input video stream is re-encoded, including: asymmetrically re-encoding the input video stream to obtain the re-encoded video stream; when the input video stream is a cascaded downstream video stream or a presentation stream, and the second attribute information of a second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded to obtain the re-encoded video stream, including: symmetrically re-encoding the input video stream to obtain the re-encoded video stream.

[0010] In an exemplary embodiment, the input video stream is symmetrically re-encoded to obtain the re-encoded video stream, including: obtaining a first identity identifier of the input video; re-encoding the input video stream to obtain the re-encoded video stream; and setting the identity identifier of the re-encoded video stream to the first identity identifier.

[0011] In an exemplary embodiment, the input video stream is asymmetrically re-encoded to obtain the re-encoded video stream, including: obtaining a first identity identifier of the input video; re-encoding the input video stream to obtain the re-encoded video stream; and setting the identity identifier of the re-encoded video stream to a second identity identifier, wherein the first identity identifier is different from the second identity identifier.

[0012] In an exemplary embodiment, the input video stream is re-encoded according to the attribute information of each terminal to obtain a re-encoded video stream, and it also includes: using a multi-screen fusion module of a conference system to re-encode the input video stream, and the conference system is a software system for the video conference.

[0013] In an exemplary embodiment, the identity of the re-encoded video stream is set to a second identity, including: obtaining the identity of a transcoding simulation terminal, and determining the second identity based on the identity of the transcoding simulation terminal, wherein the transcoding simulation terminal is a dynamically generated virtual terminal; after the asymmetric re-encoding of the input video stream, the method further includes: determining the input video that has not been re-encoded as a filler screen; inputting the filler screen into the multi-screen fusion layout of the video conference to generate a multi-screen fusion video stream.

[0014] According to another embodiment of the present invention, a device for sending a video stream is provided, comprising: an acquisition module for acquiring attribute information of each of a plurality of terminals, wherein the attribute information comprises first attribute information and second attribute information, the first attribute information being used to indicate whether the video stream sent by the terminal needs to be re-encoded, and the second attribute information being used to indicate whether the video stream received by the terminal is a re-encoded video stream, and the plurality of terminals participate in the same video conference; a re-encoding module being used to re-encode the input video stream according to the attribute information of each of the terminals to obtain a re-encoded video stream; and a sending module being used to send the input video stream or the re-encoded video stream to some or all of the plurality of terminals according to the attribute information of each of the terminals.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0016] According to another embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0017] Through the present invention, attribute information of each terminal among multiple terminals is obtained, wherein the attribute information includes first attribute information and second attribute information, the first attribute information is used to indicate whether the video stream sent by the terminal needs to be re-encoded, and the second attribute information is used to indicate whether the video stream received by the terminal is a re-encoded video stream; the input video stream is re-encoded according to the attribute information of each terminal to obtain a re-encoded video stream; according to the attribute information of each terminal, the input video stream or the re-encoded video stream is sent to multiple terminals.

[0018] Since the first attribute information in the attribute information is used to indicate whether the video stream sent by the terminal needs to be re-encoded, and the second attribute information is used to indicate whether the video stream received by the terminal is a re-encoded video stream, by obtaining the first attribute information of each terminal among multiple terminals, the input video stream can be re-encoded according to the terminal type of each terminal to obtain the re-encoded video stream, and the input video stream or the re-encoded video stream can be sent to multiple terminals according to the second attribute information of each terminal. Even if multiple terminals have the problem of poor quality of the received video stream, the sent video stream is re-encoded according to the attribute information of the terminal, so that the terminal can receive the re-encoded video stream and play it. Therefore, the problem of poor quality of the video stream received by the terminal due to the compatibility of the video stream code stream in the prior art can be solved, and the compatibility of the video code stream between the old and new terminals is met while ensuring that the new terminal can receive the delayed video stream with unchanged quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a hardware structure block diagram of a mobile terminal according to a method for sending a video stream according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of a method for sending a video stream according to an embodiment of the present invention;

[0021] Figure 3 is a structural block diagram of a video conferencing system according to an embodiment of the present invention;

[0022] Figure 4 4 is a structural block diagram of a device for sending a video stream according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] A video conferencing system consists of an MCU (Multipoint Control Unit), also known as a video conferencing server, as well as various participating devices such as software and hardware terminals and recording and broadcasting servers. The terminals capture images and audio, encode them, and send them to the MCU. The MCU then fuses or unfurls the images sent by multiple terminals and mixes or unmixes the audio, depending on the conference requirements, before sending it to each participating terminal, enabling audio and video calls for multiple participants.

[0024] However, making video conferencing systems compatible with a variety of new and old terminals becomes a difficult technical issue, especially the compatibility of video streams. Even if all encoding parameters are consistent and the signaling encoding capability set negotiation is successful, the video streams output by encoders on different terminals may still be incompatible. Video streams sent by terminals from certain manufacturers may not be received and played by all terminals. For example, when terminal A of brand X and terminal B of brand Y negotiate the video capability set, they claim to support H264BP, 25fps, and 720p. The conference encoding parameters set when creating the conference also use this video encoding format. When A and B forward the video to each other, due to the different encoding details such as the number of slices, slice size, and number of reference frames implemented by the encoder, terminal A still cannot properly receive and decode the video stream sent by terminal B.

[0025] Currently, video stream playback primarily utilizes a "symmetric identity" transcoding method, where the video is transcoded and then fed into a "visual network server" for distribution. This results in all visual network terminals on the server being restricted to using the transcoded video stream, increasing delays and degrading image quality for the target image seen by new visual network terminals. Furthermore, the compatibility issues associated with video streams when encoding formats are negotiated remain unresolved. Furthermore, this method requires an additional transcoding server cluster for transcoding calculations, resulting in high system complexity and poor maintainability.

[0026] This application aims to solve the problem that the video code streams of various new and old terminals in a newly built video conferencing system are incompatible, resulting in poor quality of the video streams transmitted to each other, and provides a method for sending video streams in order to solve the above problem.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with embodiments.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal for a method of sending a video stream according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0030] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the method for transmitting a video stream in the embodiment of the present invention. The processor 102 executes the computer program stored in the memory 104 to execute various functional applications and data processing, thereby implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0031] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the mobile terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a method for sending a video stream running on the above mobile terminal is provided. Figure 2 is a flow chart of a method for sending a video stream according to an embodiment of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0033] Step S202: Acquire attribute information of each of the multiple terminals, wherein the attribute information includes first attribute information and second attribute information, the first attribute information being used to indicate whether a video stream sent by the terminal requires re-encoding, and the second attribute information being used to indicate whether a video stream received by the terminal is a re-encoded video stream, and the multiple terminals are participating in the same video conference;

[0034] The above attribute information is used to configure the terminal type of the terminal. For example, the terminal type for which the uplink video stream needs to be transcoded (i.e., re-encoded) can be configured as "terminal type I", and the terminal type for which the uplink video stream of "terminal type I" needs to be transcoded downstream to output the video stream can be set to "terminal type II", etc. The terminal can be a device with video playback function, such as a mobile phone, tablet, computer, smart TV, etc.

[0035] Step S204, re-encoding the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream;

[0036] The attribute information of the terminal is first attribute information. The system re-encodes the input video stream accordingly according to the first attribute information of each terminal to obtain a re-encoded video stream.

[0037] The system adds the interface DB_AddVideoTranscodingDevices of the video transcoding simulation terminal to implement video stream input. The input parameters include the conference number and the transcoding input video stream InSSRC, etc. At the same time, through SetVideoTranscodingLayout, the video stream is re-encoded according to the first attribute information.

[0038] Specifically, re-encoding the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream includes:

[0039] When the input video stream is a video stream sent by a first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, re-encoding the input video stream to obtain the re-encoded video stream;

[0040] The first terminal may be one of the multiple terminal devices. When the first attribute information of the first terminal indicates that the input video stream needs to be re-encoded, the input video stream is re-encoded.

[0041] As an optional implementation, when the input video stream is a video stream sent by the first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, the input video stream is re-encoded, including: asymmetrically re-encoding the input video stream to obtain the re-encoded video stream.

[0042] The system determines whether to use symmetric or asymmetric transcoding based on the type of the transcoded input video stream InSSRC. For example, asymmetric transcoding is used for the terminal mainstream, while symmetric transcoding is used for the H323 cascaded downlink video stream and presentation stream. If the input video stream is a video stream sent by the first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the input video stream needs to be re-encoded, the transcoded input video stream InSSRC is asymmetricly encoded.

[0043] Optionally, when the transcoded input video stream InSSRC is a terminal device participating in the same video stream conference, the system adopts an asymmetric transcoding method, adds a transcoding simulation terminal by calling DB_AddVideoTranscodingDevices, obtains the terminal ID of the first terminal, and asymmetrically encodes the transcoded input video stream InSSRC according to the terminal ID of each terminal in the multiple terminals receiving the video stream terminal to obtain the transcoded output video stream OutSSRC.

[0044] By adopting an asymmetric transcoding method, the fusion stream of the public multi-screen fusion layout of the conference is not affected by the transcoding requirements of the terminal video stream, avoiding the nesting of transcoding first and then fusion layout, which leads to increased delay and deterioration of image quality in the conference fusion stream, and easy deadlock of software processing.

[0045] When the input video stream is a cascaded downstream video stream or a presentation stream, and the second attribute information of the second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded to obtain the re-encoded video stream.

[0046] Among them, the second terminal can be a cascaded downlink video stream or a presentation stream, for example, an H323 cascaded downlink video stream and a presentation stream. When the second attribute information of the second terminal indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded.

[0047] As an optional implementation, when the input video stream is a cascaded downlink video stream or a presentation stream, and the second attribute information of the second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded to obtain the re-encoded video stream, including: symmetrically re-encoding the input video stream to obtain the re-encoded video stream.

[0048] The system determines whether to use symmetric or asymmetric transcoding based on the type of the transcoded input video stream InSSRC. For example, asymmetric transcoding is used for the terminal's mainstream video stream, while symmetric transcoding is used for the H323 cascaded downlink video stream and presentation stream. If the input video stream is an H323 cascaded downlink video stream and presentation stream, and the second attribute information of the second terminal indicates that the video stream received by the second terminal is a re-encoded video stream, the transcoded input video stream InSSRC is symmetricly encoded.

[0049] Optionally, when the transcoded input video stream InSSRC is a pre-recorded video, that is, when the input video stream is a cascaded downstream video stream or a presentation stream, the system adopts a symmetric transcoding method and directly replaces the transcoded input video stream InSSRC with the transcoded output video stream OutSSRC.

[0050] Depending on the type of transcoding input video stream, you can flexibly configure "symmetric transcoding" or "asymmetric transcoding" for different video streams to meet various video stream transcoding requirements.

[0051] Step S206: Send the input video stream or the re-encoded video stream to some or all of the multiple terminals according to the attribute information of each terminal.

[0052] Specifically, when some of the multiple terminals are receiving ends of the input video stream, the input video stream or the re-encoded video stream is sent to some of the multiple terminals;

[0053] When the input video stream is a video stream sent by the first terminal among multiple terminals, that is, some of the multiple terminals are receiving ends of the input video stream, the re-encoded video stream is sent to some or all of the multiple terminals.

[0054] In a case where all of the multiple terminals are receiving ends of the input video stream, the input video stream or the re-encoded video stream is sent to all of the multiple terminals.

[0055] When the input video stream is a cascaded downlink video stream or a presentation stream, that is, all of the multiple terminals are receiving ends of the input video stream, the re-encoded video stream is sent to some or all of the multiple terminals.

[0056] By sending the corresponding transcoded output video stream according to the attribute information of different terminals, it is possible to flexibly configure that transcoding is triggered when a terminal of a specific manufacturer and model is selected for viewing. The terminal of a specific manufacturer and model is configured to download the transcoded video stream, while other types of terminals download the original video stream. This solves the compatibility problem of video stream code streams between terminals of multiple manufacturers and models when the negotiation capability set is passed, ensuring that the picture received by the new terminal still has the "low latency and high image quality" effect before transcoding.

[0057] Optionally, the execution entity of the above steps can be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone and other terminals, but is not limited to this.

[0058] Through the above steps, since the first attribute information in the attribute information is used to indicate whether the video stream sent by the terminal needs to be re-encoded, and the second attribute information is used to indicate whether the video stream received by the terminal is a re-encoded video stream, by obtaining the first attribute information of each terminal in the multiple terminals, the input video stream can be re-encoded according to the terminal type of each terminal to obtain the re-encoded video stream, and the input video stream or the re-encoded video stream can be sent to the multiple terminals according to the second attribute information of each terminal. Even if the multiple terminals have the problem of poor quality of the received video stream, the sent video stream is re-encoded according to the attribute information of the terminal, so that the terminal can receive the re-encoded video stream and play it. Therefore, the problem of poor quality of the video stream received by the terminal due to the compatibility problem of the video stream code stream in the prior art can be solved, and the effect of low delay and high image quality of the video received between terminals with incompatible video code streams can be achieved when the negotiation capability set is passed.

[0059] By adding a transcoding simulation terminal to trigger the request for the video stream to be transcoded, the uplink of the corresponding original video stream of the terminal to be transcoded is triggered, and the transcoded output video stream is input into the uplink data processing module of the media service as the uplink video stream of the transcoding simulation terminal, just like a new terminal is added to the meeting, realizing the asymmetric transcoding mode of the video stream in the meeting. Since the unique identification of the transcoding input and output streams realized by this mechanism is asymmetric, the media service can realize the downlink pre-transcoding or post-transcoding video stream for different types of terminals, and at the same time, the video stream of the multi-screen fusion layout common to the conference is input as the pre-transcoding video stream. It solves the compatibility problem of video stream code streams between specific types of reused terminals when the negotiation capability set is passed, and at the same time, it does not affect the conference screen effect of forwarding single streams and fused streams of new model terminals.

[0060] In addition, this application directly utilizes the encoding and decoding capabilities of the video conferencing server MCU for transcoding, without the need to deploy an additional transcoding server, and the overall system operation and maintenance complexity is low.

[0061] As an optional implementation, the input video stream is symmetrically re-encoded to obtain the re-encoded video stream, including: obtaining a first identity identifier of the input video; re-encoding the input video stream to obtain the re-encoded video stream; and setting the identity identifier of the re-encoded video stream to the first identity identifier.

[0062] The first identity identifier is used to indicate the current transcoding status of the input video stream, including not transcoded, transcoded, etc.

[0063] For example, the first identity identifier of the transcoded input video stream curInSSRC is InSSRC, the first identity identifier of the current input video to be transcoded is obtained and the current input video stream to be transcoded is re-encoded according to the type of the current input video to be transcoded to obtain the re-encoded video stream curOutSSRC (that is, the first video stream), and the identity identifier of the re-encoded video stream curOutSSRC is set to the first identity identifier InSSRC.

[0064] As an optional implementation, the input video stream is asymmetrically re-encoded to obtain the re-encoded video stream, including: obtaining a first identity identifier of the input video; re-encoding the input video stream to obtain the re-encoded video stream; setting the identity identifier of the re-encoded video stream to a second identity identifier, wherein the first identity identifier is different from the second identity identifier.

[0065] The first identity identifier and the second identity identifier are used to indicate the current transcoding status of the input video stream, including not transcoded, transcoded, etc. The first identity identifier is different from the second identity identifier.

[0066] For example, the first identity identifier of the current input video stream to be transcoded curInSSRC is InSSRC. The first identity identifier of the current input video to be transcoded is obtained and the current input video stream to be transcoded is encoded according to the type of the current input video stream to be transcoded to obtain the current transcoded output video stream curOutSSRC (that is, the re-encoded video stream), and the identity identifier of the current transcoded output video stream curOutSSRC is set to the second identity identifier OutSSRC.

[0067] As an optional implementation method, the input video stream is re-encoded according to the attribute information of each terminal to obtain a re-encoded video stream, and it also includes: using the multi-screen fusion module of the conference system to re-encode the input video stream, and the conference system is the software system of the video conference.

[0068] The multi-view fusion module in the video conferencing software system provides a configuration interface for the multi-view fusion layout of a video fusion session, such as the SetVideoTranscodingLayoutMS interface, which enables the opening, modification, and closing of a fusion session for transcoding purposes. Parameters for the multi-view fusion layout include the input video stream InSSRC to be transcoded, the output video stream OutSSRC to be transcoded, and encoding parameters such as the output video stream resolution and bitrate for the single-view fusion layout used for transcoding.

[0069] By determining whether this setting includes the parameter configuration of the transcoded output video stream OutSSRC encoding, the transcoding status of the current video is marked, including: if the parameter configuration of the transcoded output video stream OutSSRC encoding is not included, executing the fusion session closing operation, setting the current transcoded video stream receiving socket reference curTranscodingSocket to be transcoded to null, setting the current transcoded input video stream curInSSRC to be transcoded to 0, setting the current transcoded output video stream curOutSSRC to 0, calling the video fusion module to close the fusion session interface, closing the fusion session for transcoding purposes, and marking the current video transcoding status as "not transcoded"; if the parameter configuration of the transcoded output video stream OutSSRC encoding is included, determining the current video transcoding status, if the transcoding status is "not transcoded", calling the video fusion module to open the fusion session interface to open the fusion session for transcoding purposes, and marking the current video transcoding status as "transcoding in progress".

[0070] Determine whether the new transcoding input video stream InSSRC to be transcoded is equal to the current transcoding input video stream curInSSRC to be transcoded. If they are not equal, that is, the transcoding input video stream to be transcoded has been switched, set the current transcoding video stream receiving socket reference curTranscodingSocket to be null, set the current transcoding input video stream curInSSRC to InSSRC, and set the current transcoding output video stream curOutSSRC to OutSSRC.

[0071] Finally, by calling the configuration interface of the multi-screen fusion layout of the video fusion module, the latest fusion layout and fusion encoding parameters are set to the video fusion module, so that it uses the latest InSSRC as the input video stream and outputs the transcoded OutSSRC video stream, thereby realizing the multi-screen fusion module of the conference system to re-encode the input video stream.

[0072] As an optional implementation, setting the identity of the re-encoded video stream as a second identity includes: obtaining the identity of a transcoding simulation terminal, and determining the second identity based on the identity of the transcoding simulation terminal, wherein the transcoding simulation terminal is a dynamically generated virtual terminal.

[0073] Use transcoding simulation terminal logic to implement asymmetric transcoding of physical terminal video streams. This involves calling DB_AddVideoTranscodingDevices, dynamically adding a transcoding simulation terminal in the SetVideoTranscodingLayout interface, and obtaining the terminal ID of the simulation terminal. A transcoding simulation terminal is a globally unique virtual terminal dynamically generated based on the asymmetric transcoding requirements of the conference video stream. The transcoded output video stream OutSSRC is generated based on the terminal ID, simulating the transcoded output video stream as the uplink video stream of the transcoding simulation terminal. For example, if the terminal ID of the transcoding simulation terminal is 16000001, the re-encoded transcoded output video stream OutSSRC is 160000012.

[0074] As an optional implementation, after the asymmetric re-encoding of the input video stream, the method further includes: determining the input video that has not been re-encoded as a filler screen; inputting the filler screen into the multi-screen fusion layout of the video conference to generate a multi-screen fusion video stream.

[0075] After the asymmetric re-encoding of the input video stream, the unre-encoded input video is input as the terminal uplink video stream to the uplink data processing module of the media service. The uplink data processing module then inputs the unre-encoded input video into the fusion session of the public multi-screen fusion layout as needed according to the conference layout (that is, the video stream is input only if it is filled in the fusion layout, otherwise it is not input), and distributes it to each participant terminal that needs the uplink video stream through network IO.

[0076] Based on asymmetric transcoding, the input video stream that has not been re-encoded is input as a filler screen into the multi-screen fusion layout of the conference to generate a multi-screen fusion video stream, ensuring that the delay of the multi-screen fusion video stream does not increase.

[0077] As an optional implementation, Figure 3 FIG is a structural block diagram of a video conferencing system according to an embodiment of the present invention. Figure 3As shown, the video conferencing system provides an interface for opening and closing video fusion sessions, an interface for configuring the multi-screen fusion layout of video fusion sessions, and an interface for inputting video streams to video fusion sessions. It mainly includes conference control services, signaling services, media services, etc., and implements video fusion processes such as input video stream decoding, scaling, layout filling and splicing, and encoding, and finally provides a callback interface to output the fused video stream.

[0078] Conference control service logic: Provides an interface to implement terminal type configuration. The terminal type for which the upstream video stream needs to be transcoded is configured as "Terminal Type I". The terminal type for which the upstream video stream selected to view "Terminal Type I" needs to be transcoded downstream to output the video stream is set to "Terminal Type II", thereby implementing the transcoding switch for the upstream video stream of "Terminal Type I".

[0079] Signaling service logic:

[0080] Implement the interface DB_AddVideoTranscodingDevices for adding video transcoding simulation terminals. The parameters include the conference number and the transcoding input video stream InSSRC. Adding triggers the logic: Step 311, generate the globally unique terminal number devid for the simulation terminal; Step 312, specify the simulation terminal type as a specific type of transcoding simulation terminal, such as 1602; Step 313, specify the simulation terminal status as having joined the corresponding conference; Step 314, initiate a video stream request RequestStreams in the name of the terminal, requesting the uplink transcoding input video stream InSSRC of the corresponding terminal to be transcoded.

[0081] Implement the DB_DeleteVideoTranscodingDevices interface for deleting a video transcoding simulation terminal: Step 321: Initiate a RequestStreams request on behalf of the simulation terminal to request an empty video stream so that the corresponding terminal stops uplink transcoding the input video stream InSSRC; Step 322: Delete the simulation terminal from the device list;

[0082] Implement the SetVideoTranscodingLayout interface for starting, modifying, and shutting down video transcoding. Parameters include the conference number, the transcoding input video stream InSSRC, and whether transcoding is enabled. Step 331: Determine whether symmetric or asymmetric transcoding is used based on the InSSRC type. For example, the terminal mainstream uses asymmetric transcoding, while the H323 cascaded downlink video stream and presentation stream use symmetric transcoding. If it is asymmetric transcoding, call DB_AddVideoTranscodingDevices to add a transcoding simulation terminal, obtain the terminal ID, and generate the transcoding output video stream OutSSRC based on the terminal ID. This means simulating the transcoding output video stream to the uplink video stream of the transcoding simulation terminal. If it is symmetric transcoding, then set the transcoding output video stream OutSSRC to the transcoding input video stream InSSRC. This means replacing the transcoding input video stream with the transcoding output video stream, while the identity of the video stream remains unchanged. Step 332: Determine whether transcoding is enabled. If transcoding is disabled, call DB_DeleteVideoTranscodingDevices to delete the transcoding simulation terminal, set the transcoding input video stream InSSRC to empty, set the transcoding output video stream OutSSRC to empty, set the transcoding encoding parameters to empty, and call the SetVideoTranscodingLayoutMS interface of the media service to disable transcoding. If transcoding is enabled, use the InSSRC, OutSSRC, and preset transcoding video encoding parameters set above to generate a fusion layout for transcoding purposes, and call the SetVideoTranscodingLayoutMS interface of the media service to enable or modify transcoding.

[0083] The forwarding control operation implementation logic such as "Video Stream Selection" determines that the preset terminal type I video stream transcoding switch is turned on, then executes: Step 341, if the video stream selected by "Video Stream Selection" is the upstream video stream mainstream of the preset terminal type I, then call the SetVideoTranscodingLayout interface of the signaling service for the video stream to enable transcoding or modify transcoding, and modify transcoding to achieve switching of the transcoded video stream; Step 342, if the video stream selected by "Video Stream Selection" is the video stream of other types of terminals, then call the SetVideoTranscodingLayout interface of the signaling service to turn off transcoding; Step 343, determine that the terminal type is the preset terminal type II, and its video stream selected by "Video Stream Selection" is the transcoding input video If the stream InSSRC is empty and the transcoded output video stream OutSSRC is not empty, the terminal selects the video stream SSRC to be replaced with the current transcoded output video stream OutSSRC; the terminal of the preset terminal type II requests the transcoded video stream, while other types of terminals still request the video stream InSSRC before transcoding; a "video stream selection" operation is implemented to output two video streams for different types of terminals; step 344, when it is detected that the media service is restarted, if there is a fusion session for transcoding purposes, SetVideoTranscodingLayout is called to re-trigger transcoding; step 345, the conference encoding resource overhead is calculated, if the conference transcoding input video stream InSSRC is not empty, the fusion layout encoding and decoding overhead for transcoding purposes is added.

[0084] Media service logic:

[0085] Implement the converged layout setting interface SetVideoTranscodingLayoutMS for video transcoding, which enables the opening, modification, and closing of converged sessions for transcoding. The specified parameters include: the single-screen converged layout for transcoding, the input video stream to be transcoded inSSRC, the transcoded output video stream outSSRC, and encoding parameters such as the output video stream resolution and bit rate.

[0086] Step 351: Determine whether the current setting includes the output video stream OutSSRC encoding parameter configuration:

[0087] If not, the fusion session is closed. The reference to the receiving socket for the current video stream to be transcoded, curTranscodingSocket, is cleared. The current input video stream to be transcoded, curInSSRC, is set to 0. The current output video stream to be transcoded, curOutSSRC, is set to 0. The video fusion module is called to close the fusion session interface, closing the fusion session for transcoding purposes. The current video transcoding status is marked as "not transcoded." This completes the SetVideoTranscodingLayoutMS interface setup.

[0088] If included, the current video transcoding status is determined. If it is "not transcoded", the video fusion module is called to open the fusion session interface to open the fusion session for transcoding purposes; the current video transcoding status is marked as "transcoding".

[0089] Step 352: Determine whether the new video stream to be transcoded InSSRC is equal to the current video stream to be transcoded curInSSRC:

[0090] If they are not equal, it means that the video stream to be transcoded has been switched, and the reference to the receiving socket of the current video stream to be transcoded, curTranscodingSocket, is set to null; the current input video stream to be transcoded, curInSSRC, is set to InSSRC; the current output video stream to be transcoded, curOutSSRC, is set to OutSSRC;

[0091] Step 353: Call the multi-image fusion layout configuration interface of the video fusion module, set the latest fusion layout and fusion encoding parameters to the video fusion module, so that it uses the latest InSSRC as the input video stream and outputs the transcoded OutSSRC video stream;

[0092] The network IO data callback interface of the receiving terminal audio and video stream implements the logic to determine that the transcoding status is "transcoding" and the SSRC of the currently received video stream is the current input video stream to be transcoded curInSSRC:

[0093] Step 361: If yes, call the video stream input interface of the video fusion module to input the callback video stream data into the fusion session for transcoding; determine whether the current input video stream to be transcoded curInSSRC is equal to the current transcoded output video stream curOutSSRC:

[0094] If not, the callback video stream data is input to the uplink data processing module of the media service. The uplink data processing module then inputs the callback video stream data into the fusion session of the public multi-screen fusion layout as needed according to the conference layout (that is, it is input only if the fusion layout is filled with the video stream ssrc, otherwise it is not input), and distributes it to each participant terminal that needs the uplink video stream through network IO;

[0095] If they are equal, the current transcoded video stream receiving socket reference curTranscodingSocket is set to the client socket socket corresponding to this network IO callback; the callback video stream data is not input into the uplink data processing module of the media service;

[0096] Step 362: If not, the callback data is input into the uplink data processing module of the media service.

[0097] The logic implemented in the fusion data callback interface of the fusion session for transcoding is as follows: Determine the curInSSRC value of the current input video stream to be transcoded:

[0098] Step 371, if it is equal to 0, then the callback data of this converged session is ignored and discarded;

[0099] Step 372, if not equal to 0, determines whether the current transcoded video stream receiving socket reference curTranscodingSocket is empty:

[0100] If it is empty, the callback video stream of this converged session is input into the converged data processing module of the media service, so that it can be distributed to each participant terminal that needs the converged video stream through the network IO;

[0101] If it is not empty, the callback video stream data of this fusion session will be used as the terminal uplink video stream and input into the uplink data processing module of the media service. The uplink data processing module will input the callback video stream data into the fusion session of the public multi-screen fusion layout as needed according to the conference layout (that is, it will be input only if the fusion layout is filled with the video stream ssrc, otherwise it will not be input), and distribute it to each participant terminal that needs the uplink video stream through network IO.

[0102] This application utilizes the encoding and decoding capabilities of the video conferencing server MCU to transcode the video stream, and triggers the request for the video stream to be transcoded by adding a transcoding simulation terminal, thereby triggering the uplink of the corresponding original video stream of the terminal to be transcoded, and inputting the transcoded output video stream as the uplink video stream of the transcoding simulation terminal into the uplink data processing module of the media service. The media service can realize the downlink video stream before or after transcoding for different types of terminals, and realize that the fusion stream of the conference public multi-screen fusion layout is not affected by the terminal video stream transcoding requirements, avoiding the problems of nested fusion layout after transcoding first and then fusion, increased delay of conference fusion stream and deterioration of image quality, and easy deadlock of software processing. Therefore, the problem of video code stream compatibility in the existing technology can be solved, and the effect of ensuring low delay and high image quality of the video picture after multi-screen fusion is achieved.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0104] In this embodiment, a device for transmitting a video stream is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments. Details already described are omitted for clarity. As used below, the term "module" may refer to a combination of software or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0105] Figure 4 is a structural block diagram of a device for sending a video stream according to an embodiment of the present invention, such as Figure 4 As shown, the device includes: an acquisition module 42, used to obtain attribute information of each of a plurality of terminals, wherein the attribute information includes first attribute information and second attribute information, the first attribute information is used to indicate whether the video stream sent by the terminal needs to be re-encoded, and the second attribute information is used to indicate whether the video stream received by the terminal is a re-encoded video stream, and the plurality of terminals participate in the same video conference; a re-encoding module 44, used to re-encode the input video stream according to the attribute information of each of the terminals to obtain a re-encoded video stream; a sending module 46, used to send the input video stream or the re-encoded video stream to some or all of the plurality of terminals according to the attribute information of each of the terminals.

[0106] As an optional embodiment, the device is also used to re-encode the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream, including: when the input video stream is a video stream sent by the first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, re-encoding the input video stream to obtain the re-encoded video stream; when the input video stream is a cascaded downstream video stream or a presentation stream, and the second attribute information of the second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, re-encoding the input video stream to obtain the re-encoded video stream.

[0107] As an optional embodiment, the device is also used to re-encode the input video stream when the input video stream is a video stream sent by the first terminal among the multiple terminals and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, including: symmetrically re-encoding the input video stream to obtain the re-encoded video stream; when the input video stream is a cascaded downstream video stream or a presentation stream and the second attribute information of the second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, re-encoding the input video stream to obtain the re-encoded video stream, including: asymmetrically re-encoding the input video stream to obtain the re-encoded video stream.

[0108] As an optional embodiment, the device is also used to symmetrically re-encode the input video stream to obtain the re-encoded video stream, including: obtaining a first identity identifier of the input video; re-encoding the input video stream to obtain the re-encoded video stream; and setting the identity identifier of the re-encoded video stream to the first identity identifier.

[0109] As an optional embodiment, the device is also used to asymmetrically re-encode the input video stream to obtain the re-encoded video stream, including: obtaining a first identity identifier of the input video; re-encoding the input video stream to obtain the re-encoded video stream; setting the identity identifier of the re-encoded video stream to a second identity identifier, wherein the first identity identifier is different from the second identity identifier.

[0110] As an optional implementation, the apparatus is further configured to use a multi-image fusion module of a conference system to re-encode the input video stream, and the conference system is a software system for the video conference.

[0111] As an optional embodiment, the apparatus is further configured to set the identity of the re-encoded video stream as a second identity, comprising: obtaining an identity of a transcoding simulation terminal, and determining the second identity according to the identity of the transcoding simulation terminal, wherein the transcoding simulation terminal is a dynamically generated virtual terminal;

[0112] As an optional embodiment, the device is also used to, after the asymmetric re-encoding of the input video stream, determine the input video that has not been re-encoded as a filler screen; input the filler screen into the multi-screen fusion layout of the video conference to generate a multi-screen fusion video stream.

[0113] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0114] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0115] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0116] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0117] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0118] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0119] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0120] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for sending a video stream, characterized in that: include: Obtaining attribute information of each of the multiple terminals, wherein the attribute information includes first attribute information and second attribute information, the first attribute information being used to indicate whether a video stream sent by the terminal needs to be re-encoded, and the second attribute information being used to indicate whether a video stream received by the terminal is a re-encoded video stream, the multiple terminals participating in the same video conference; re-encoding the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream, wherein the input video stream and the re-encoded video stream have the same encoding format and different encoding parameters; sending the input video stream or the re-encoded video stream to some or all of the multiple terminals according to the attribute information of each terminal; The step of sending the input video stream or the re-encoded video stream to some or all of the multiple terminals according to the attribute information of each terminal includes: Sending the input video stream to a terminal among the multiple terminals, the second attribute information indicating that the received video stream is not the re-encoded video stream; The re-encoded video stream is sent to a terminal among the multiple terminals, the second attribute information indicating that the received video stream is the re-encoded video stream.

2. The method according to claim 1, characterized in that Re-encoding the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream, including: When the input video stream is a video stream sent by a first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, re-encoding the input video stream to obtain the re-encoded video stream; When the input video stream is a cascaded downstream video stream or a presentation stream, and the second attribute information of the second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded to obtain the re-encoded video stream.

3. The method according to claim 2, characterized in that When the input video stream is a video stream sent by a first terminal among the multiple terminals, and the first attribute information of the first terminal indicates that the video stream sent by the first terminal needs to be re-encoded, re-encoding the input video stream includes: asymmetrically re-encoding the input video stream to obtain the re-encoded video stream; When the input video stream is a cascaded downstream video stream or a presentation stream, and the second attribute information of the second terminal among the multiple terminals indicates that the video stream received by the second terminal is a re-encoded video stream, the input video stream is re-encoded to obtain the re-encoded video stream, including: symmetrically re-encoding the input video stream to obtain the re-encoded video stream.

4. The method according to claim 3, characterized in that Symmetrically re-encoding the input video stream to obtain the re-encoded video stream includes: Obtaining a first identity identifier of the input video; Re-encoding the input video stream to obtain the re-encoded video stream; The identity of the re-encoded video stream is set as the first identity.

5. The method according to claim 3, characterized in that Asymmetrically re-encoding the input video stream to obtain the re-encoded video stream includes: Obtaining a first identity identifier of the input video; Re-encoding the input video stream to obtain the re-encoded video stream; The identity of the re-encoded video stream is set as a second identity, wherein the first identity is different from the second identity.

6. The method according to claim 1, characterized in that Re-encoding the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream, further comprising: The input video stream is re-encoded using a multi-image fusion module of a conference system, where the conference system is a software system for the video conference.

7. The method according to claim 5, characterized in that Setting the identity of the re-encoded video stream as a second identity includes: obtaining an identity of a transcoding simulation terminal, and determining the second identity according to the identity of the transcoding simulation terminal, wherein the transcoding simulation terminal is a dynamically generated virtual terminal; After the asymmetric re-encoding of the input video stream, the method further includes: determining the input video that has not been re-encoded as a filler screen; inputting the filler screen into the multi-screen fusion layout of the video conference to generate a multi-screen fusion video stream.

8. A video stream sending device, characterized in that: include: an acquisition module, configured to acquire attribute information of each of a plurality of terminals, wherein the attribute information includes first attribute information and second attribute information, the first attribute information being used to indicate whether a video stream sent by the terminal requires re-encoding, and the second attribute information being used to indicate whether a video stream received by the terminal is a re-encoded video stream, and the plurality of terminals participating in the same video conference; a re-encoding module, configured to re-encode the input video stream according to the attribute information of each terminal to obtain a re-encoded video stream, wherein the encoding format of the input video stream and the re-encoded video stream is the same, and the encoding parameters of the input video stream and the re-encoded video stream are different; a sending module, configured to send the input video stream or the re-encoded video stream to some or all of the multiple terminals according to the attribute information of each terminal; The device is also used to send the input video stream to the terminal among the multiple terminals, where the second attribute information indicates that the received video stream is not the re-encoded video stream; and send the re-encoded video stream to the terminal among the multiple terminals, where the second attribute information indicates that the received video stream is the re-encoded video stream.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images