Communication method, system and device based on virtual convergence gateway, and storage medium

Abstract

The embodiment of the invention discloses a communication method, system and device based on a virtual convergence gateway and a storage medium, video stream transmission in the whole connection process is that a network is detected through webtc so as to control transmission of video streams, and the video quality is ensured; a plurality of video sources use different communication protocols, video data are accessed into the same conference, and different video formats can be decoded into a unified format, so that the video formats and the communication protocols of the video sources are not required to be unified, and the compatibility is higher; the soft terminal is driven by calling the virtual camera through the browser, so that the bandwidth can be saved under the condition of the standard browser; the multi-channel video streams are locally synthesized into a single-channel video stream and then the single-channel video stream is sent to remote equipment to reduce the occupied bandwidth; the layout of the image displayed by the virtual camera can be changed at any time, so that a user has more comfortable viewing experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video communication technology, and in particular to a communication method, system, device and storage medium based on a virtual fusion gateway. Background Art

[0002] The converged gateway is a key node that connects different networks and realizes the conversion of multiple communication protocols and data exchange. The converged gateway not only realizes the interoperability of different networks, but also realizes seamless connection and data sharing between various devices through its efficient protocol conversion function. The virtual gateway is a physical access router and a logical access gateway that realizes physical connection access to VPC.

[0003] When users need to view images from multiple remote video terminals in real time, they need to communicate with multiple video terminals at the same time and transmit multiple video streams at the same time. Due to bandwidth issues, if a weak network is encountered, serious packet loss may occur, resulting in the received video being choppy or severely stuck. In addition, due to the existence of video devices with multiple video formats and different communication protocols, the existing video communication function has certain restrictions on the communication protocol and video format, resulting in a decrease in video quality. Summary of the invention

[0004] To this end, the embodiments of the present invention provide a communication method, system, device and storage medium based on a virtual converged gateway to solve the technical problems in the prior art of limited network bandwidth and easy packet loss when communicating with multiple video terminals at the same time.

[0005] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0006] According to a first aspect of an embodiment of the present invention, a communication method based on a virtual converged gateway is provided. The method is applied to the virtual converged gateway and includes:

[0007] S1. Obtain at least one video terminal to be invited and register the video terminal to be invited to the virtual fusion gateway in turn, and the virtual fusion gateway invites each registered video terminal to the conference in turn, so that each video terminal establishes a connection with the virtual fusion gateway;

[0008] S2, receiving a video stream, starting a decoder using a preset format and decoding the video stream in sequence, modifying the video stream into a video stream in a preset format and combining multiple video streams into a single video stream according to split-screen information;

[0009] S3, writing the single-channel video stream into a virtual camera driver file, synthesizing multiple video source data into single-channel video source data, and writing the data into a virtual camera device driver file;

[0010] S4, the soft terminal retrieves the video stream from the virtual camera, determines whether the virtual camera is connected to the remote terminal, and when the virtual camera is not connected to the remote terminal, the soft terminal reads the video stream from the virtual camera and displays the video stream on a local page;

[0011] S5. When the virtual camera is connected to a remote terminal for a video call, a corresponding negotiation format is obtained and the video stream is encoded using the negotiation format, and the encoded video stream is sent to the remote terminal.

[0012] Furthermore, the preset format is YUV420 format.

[0013] Furthermore, the method further comprises:

[0014] Determining whether there is a decoding requirement according to the single-channel video stream;

[0015] If there is a decoding requirement, the single-channel video stream is decoded according to the corresponding requirement and the decoded single-channel video stream is written to the virtual camera driver file;

[0016] If there is no decoding requirement, the single-channel video stream is written to the virtual camera driver file.

[0017] Furthermore, when the virtual camera is connected to a remote terminal for a video call, a corresponding negotiation format is obtained and the video stream is encoded using the negotiation format, and the encoded video stream is sent to the remote terminal, further comprising:

[0018] Determine whether there is a decoding requirement based on the corresponding video stream, soft terminal information and remote terminal information;

[0019] If there is a decoding requirement, the video stream is decoded and the corresponding negotiated format is obtained and the video stream is encoded using the negotiated format;

[0020] If there is no decoding requirement, the corresponding negotiated format is obtained and the video stream is encoded using the negotiated format.

[0021] Furthermore, the communication protocol used by the virtual camera to connect to the remote terminal for a video call is the webrtc streaming media protocol.

[0022] According to a second aspect of an embodiment of the present invention, a communication system based on a virtual converged gateway is provided, the system comprising:

[0023] A registration module, used to obtain at least one video terminal to be invited and register the video terminal to be invited to the virtual fusion gateway in turn, and the virtual fusion gateway invites each registered video terminal to the conference in turn, so that each video terminal establishes a connection with the virtual fusion gateway;

[0024] A synthesis module, used for receiving a video stream, starting a decoder using a preset format and decoding the video stream in sequence, modifying the video stream into a video stream in a preset format and synthesizing multiple video streams into a single video stream according to split-screen information;

[0025] A virtual device module, used for writing the single-channel video stream into a virtual camera driver file, synthesizing multiple video source data into single-channel video source data, and writing the data into a virtual camera device driver file;

[0026] A calling module, used for the soft terminal to call the video stream from the virtual camera, determine whether the virtual camera is connected to the remote terminal, and when the virtual camera is not connected to the remote terminal, the soft terminal reads the video stream from the virtual camera and displays the video stream on a local page;

[0027] The video call connection module is used to obtain the corresponding negotiation format and use the negotiation format to encode the video stream when the virtual camera is connected to the remote terminal for a video call, and send the encoded video stream to the remote terminal.

[0028] According to a third aspect of an embodiment of the present invention, a communication device based on a virtual converged gateway is provided, the device comprising: a processor and a memory;

[0029] The memory is used to store one or more program instructions;

[0030] The processor is used to run one or more program instructions to execute the steps of a communication method based on a virtual converged gateway as described in any one of the above items.

[0031] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a communication method based on a virtual converged gateway as described in any one of the above items are implemented.

[0032] The embodiments of the present invention have the following advantages:

[0033] The video stream transmission in the entire connection process of the embodiment of the present invention is to detect the network through webrtc to control the transmission of the video stream, thereby ensuring the quality of the video; multiple video sources are connected together in a conference manner, and can communicate through different protocols, and different video formats can be decoded into a unified format, so there is no unified requirement for the video format and communication protocol of the video source, thereby having stronger compatibility; the soft terminal calls the virtual camera driver through the browser, which can save bandwidth in the case of a standard browser; multiple video streams are synthesized into a single video stream locally and then sent to the remote device to reduce bandwidth occupation; the image layout displayed by the virtual camera can be changed at any time, so that users have a more comfortable viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0035] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0036] Figure 1 A schematic diagram of the logical structure of a communication system based on a virtual converged gateway provided in an embodiment of the present invention;

[0037] Figure 2 A flow chart of a communication method based on a virtual converged gateway provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a video transmission solution that does not use a video fusion gateway and a communication method based on a virtual fusion gateway provided in an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a video transmission solution using a virtual video fusion gateway in a communication method based on a virtual fusion gateway provided in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a group of incoming calls from a video terminal in a communication method based on a virtual converged gateway provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of a group of incoming calls from a video terminal in a communication method based on a virtual converged gateway provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of writing image decoding YUV420 data synthesis picture into a virtual camera in a communication method based on a virtual fusion gateway provided by another embodiment of the present invention;

[0043] Figure 8 A schematic diagram of the overall structure of a video fusion gateway in a communication method based on a virtual fusion gateway provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] When users need to view images from multiple remote video terminals in real time, they need to communicate with multiple video terminals at the same time and transmit multiple video streams at the same time. Due to bandwidth issues, if a weak network is encountered, serious packet loss may occur, resulting in the received video being choppy or severely stuck. In addition, due to the existence of video devices with multiple video formats and different communication protocols, the existing video communication function has certain restrictions on the communication protocol and video format, resulting in a decrease in video quality.

[0046] In order to solve the above technical problems of limited network bandwidth and easy packet loss when communicating with multiple video terminals at the same time.

[0047] refer to Figure 1 The embodiment of the present invention discloses a communication system based on a virtual converged gateway, which includes: a registration module 1; a synthesis module 2; a virtual device module 3; a calling module 4; and a video call connection module 5.

[0048] Corresponding to the above-disclosed communication system based on a virtual converged gateway, an embodiment of the present invention further discloses a communication method based on a virtual converged gateway. The following describes in detail a communication method based on a virtual converged gateway disclosed in an embodiment of the present invention in combination with the above-described communication system based on a virtual converged gateway.

[0049] When users want to watch the video device on the remote mobile vehicle during the meeting, they can only install a gateway proxy on the mobile vehicle, such as Figure 3 As shown in FIG. 1 , when calling multiple mobile vehicle-mounted devices, the video quality will be degraded due to bandwidth issues.

[0050] refer to Figure 2 The present invention discloses a communication method based on a virtual fusion gateway, which is applied to the virtual fusion gateway and includes: S1, obtaining at least one video terminal to be invited and registering the video terminal to be invited to the virtual fusion gateway in turn, and the virtual fusion gateway invites each registered video terminal to the conference in turn, so that each video terminal establishes a connection with the virtual fusion gateway.

[0051] In order to solve the problem of limited network bandwidth and clear viewing of video devices on mobile vehicles in remote conference systems, a virtual converged gateway is installed on the remote vehicle to replace the ordinary gateway, that is, reference Figure 4 After adopting the virtual converged gateway, only one video stream needs to be sent to the remote end, saving bandwidth.

[0052] refer to Figure 5 ,The video fusion gateway pulls multiple video devices (IPC cameras, SIP terminals, H323 terminals, WebCam, etc.) into the same group through different ,protocols through the video fusion gateway and synthesizes the videos.

[0053] Register all the terminals that need to be invited to the gateway system, and the gateway system will invite all the registered video terminals to the conference. In this way, the terminal will establish a connection with the gateway system and send the video stream to the gateway system.

[0054] S2, receiving a video stream, starting a decoder using a preset format and decoding the video stream in sequence, modifying the video stream into a video stream in a preset format and combining multiple video streams into a single video stream according to split-screen information.

[0055] refer to Figure 6 , pull video devices of different protocol types into the same group through the fusion gateway. After the gateway decodes the device data, the user can customize the split-screen display and set the video to be displayed on the screen. The gateway will process the video stream according to the preset information. After the gateway receives the video stream, it starts the decoder according to the format negotiated between the gateway and the terminal, decodes the video stream, and decodes the video stream into YUV420 format. Then, multiple video streams are synthesized into a single video stream according to the split-screen information.

[0056] S3, writing the single-channel video stream into a virtual camera driver file, synthesizing multiple video source data into single-channel video source data, and writing the data into a virtual camera device driver file.

[0057] refer to Figure 7,The virtual video synthesis gateway scales a set of decoded video YUV420 video data according to the ,screen split information set by the user and the video source to be displayed on the screen, ,synthesizes a picture, and writes it to the local virtual camera ,device.

[0058] Write the processed video stream to the virtual camera driver file and simulate multiple video channels into a single camera image, which is the video source of the virtual camera.

[0059] The local virtual camera device uses Linux's V4L2Loopback software. V4L2Loopback is a Linux kernel module that allows the creation of virtual video devices and passes input streams to other applications. This allows a virtual camera to be simulated without a hardware camera. Combining the virtual camera with the video fusion gateway can send multi-channel fused videos to the remote conference through the webrtc streaming protocol.

[0060] S4, the soft terminal retrieves the video stream from the virtual camera and determines whether the virtual camera is connected to the remote terminal. When the virtual camera is not connected to the remote terminal, the soft terminal reads the video stream from the virtual camera and displays the video stream on a local page.

[0061] S5. When the virtual camera is connected to a remote terminal for a video call, a corresponding negotiation format is obtained and the video stream is encoded using the negotiation format, and the encoded video stream is sent to the remote terminal.

[0062] The soft terminal retrieves the video stream from the virtual camera driver. When the analog camera is not connected to the remote device, the soft terminal will display the video stream read from the virtual camera on the local page. When the remote end makes a video call with the local soft terminal, the soft terminal will decode the video stream (may not be transferred according to demand), then encode the video stream in the format negotiated with the remote end, and then send the encoded video stream to the remote device. In this way, the remote end can watch the images of multiple video source devices in real time at the same time.

[0063] refer to Figure 8 , write the data on the local virtual camera device, and preview it locally through a browser based on the webrtc standard. At the same time, the fusion gateway can be used as a call node to call into a remote conference. The fusion gateway can also be called in by a remote video conference. When the fusion video gateway is called in by a remote conference through the webrtc protocol, the fusion gateway will collect the data on the virtual camera and transmit it to the video conference at the same time.

[0064] Furthermore, the preset format is YUV420 format.

[0065] Furthermore, the method also includes: judging whether there is a decoding requirement based on the single-channel video stream; if there is a decoding requirement, decoding the single-channel video stream according to the corresponding requirement and writing the decoded single-channel video stream to the virtual camera driver file; if there is no decoding requirement, writing the single-channel video stream to the virtual camera driver file.

[0066] Decode the synthesized video stream (may not be converted as required), for example, into RGB format.

[0067] Furthermore, when the virtual camera is connected to a remote terminal for a video call, a corresponding negotiation format is obtained and the video stream is encoded using the negotiation format, and the encoded video stream is sent to the remote terminal. It also includes: judging whether there is a decoding requirement based on the corresponding video stream, soft terminal information and remote terminal information; if there is a decoding requirement, decoding the video stream and obtaining the corresponding negotiation format and encoding the video stream using the negotiation format; if there is no decoding requirement, obtaining the corresponding negotiation format and encoding the video stream using the negotiation format.

[0068] Furthermore, the communication protocol used by the virtual camera to connect to the remote terminal for a video call is the webrtc streaming media protocol.

[0069] Since the converged video gateway media and calls use the standard WebRTC protocol, dynamic bandwidth adjustment can be achieved during network video streaming to ensure good video quality even in weak network conditions.

[0070] When the soft terminal wants to modify the layout of the gateway's composite video, the soft terminal can send a message to modify the page layout to the gateway through the interface. After receiving the message, the gateway modifies the layout, so that the layout of the video image obtained from the virtual camera can be controlled in real time.

[0071] In addition, an embodiment of the present invention also provides a communication device based on a virtual converged gateway, the device comprising: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a communication method based on a virtual converged gateway as described in any of the above items.

[0072] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a communication method based on a virtual converged gateway as described in any one of the above items are implemented.

[0073] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0074] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0075] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0076] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0077] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0078] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0079] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0080] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A communication method based on a virtual fusion gateway, characterized in that: The method is applied to a virtual converged gateway, and comprises: S1. Obtain at least one video terminal to be invited and register the video terminal to be invited to the virtual fusion gateway in turn, and the virtual fusion gateway invites each registered video terminal to the conference in turn, so that each video terminal establishes a connection with the virtual fusion gateway; S2, receiving a video stream, starting a decoder using a preset format and decoding the video stream in sequence, modifying the video stream into a video stream in a preset format and combining multiple video streams into a single video stream according to split-screen information; S3, synthesizing the multi-channel video source data into a single-channel video source data, and writing it into the virtual camera device driver file; S4, the soft terminal retrieves the video stream from the virtual camera and determines whether the virtual camera is connected to the remote terminal. When the virtual camera is not connected to the remote terminal, the data at this time can be displayed locally. When the soft terminal establishes a call connection with the remote terminal device, the data can be transmitted to the remote terminal. S5. When the virtual camera is connected to a remote terminal for a video call, a corresponding negotiation format is obtained and the video stream is encoded using the negotiation format, and the encoded video stream is sent to the remote terminal.

2. A communication method based on a virtual converged gateway as claimed in claim 1, characterized in that: The preset format is YUV420 format.

3. A communication method based on a virtual converged gateway as claimed in claim 2, characterized in that: The method further comprises: Determining whether there is a decoding requirement according to the single-channel video stream; If there is a decoding requirement, the single-channel video stream is decoded according to the corresponding requirement and the decoded single-channel video stream is written to the virtual camera driver file; If there is no decoding requirement, the single-channel video stream is written to the virtual camera driver file.

4. A communication method based on a virtual converged gateway as claimed in claim 3, characterized in that: When the virtual camera is connected to a remote terminal for a video call, a corresponding negotiation format is obtained and a video stream is encoded using the negotiation format, and the encoded video stream is sent to the remote terminal, further comprising: Determine whether there is a decoding requirement based on the corresponding video stream, soft terminal information and remote terminal information; If there is a decoding requirement, the video stream is decoded and the corresponding negotiated format is obtained and the video stream is encoded using the negotiated format; If there is no decoding requirement, the corresponding negotiated format is obtained and the video stream is encoded using the negotiated format.

5. A communication method based on a virtual converged gateway as claimed in claim 4, characterized in that: The communication protocol used by the virtual camera to connect to the remote terminal for video calls is the webrtc streaming protocol.

6. A communication system based on a virtual converged gateway, characterized in that: The system comprises: A registration module, used to obtain at least one video terminal to be invited and register the video terminal to be invited to the virtual fusion gateway in turn, and the virtual fusion gateway invites each registered video terminal to the conference in turn, so that each video terminal establishes a connection with the virtual fusion gateway; A synthesis module, used for receiving a video stream, starting a decoder using a preset format and decoding the video stream in sequence, modifying the video stream into a video stream in a preset format and synthesizing multiple video streams into a single video stream according to split-screen information; A virtual device module, used for writing the single-channel video stream into a virtual camera driver file, synthesizing multiple video source data into single-channel video source data, and writing the data into a virtual camera device driver file; A calling module, used for the soft terminal to call the video stream from the virtual camera, determine whether the virtual camera is connected to the remote terminal, and when the virtual camera is not connected to the remote terminal, the soft terminal reads the video stream from the virtual camera and displays the video stream on a local page; The video call connection module is used to obtain the corresponding negotiation format and use the negotiation format to encode the video stream when the virtual camera is connected to the remote terminal for a video call, and send the encoded video stream to the remote terminal.

7. A communication device based on a virtual converged gateway, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a communication method based on a virtual converged gateway as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the communication method based on a virtual converged gateway as claimed in any one of claims 1 to 5 are implemented.

Images