A method, device, equipment and storage medium for audio and video intercommunication

Abstract

The embodiment of the present invention discloses an audio and video intercommunication method, device, equipment and storage medium. The method includes: responding to an audio and video intercommunication request sent by a requested client in the same local area network; after the media negotiation is passed, obtaining the requesting client that the requested client requests to intercommunication with, and detecting whether it is located in the same local area network as the requesting client; if not, when the client type of the requested client is not compatible with the target coding format, converting the collected real-time audio and video stream into an appropriate audio and video stream of the target coding format; and forwarding it to the next server that matches the requesting client. Through the above method, the problem that the audio and video cannot be intercommunication caused by the inconsistent video coding format between the requested client and the requesting client is solved. Flexible audio and video interaction between the requested client and the requesting client can be achieved, expanding the applicable scenarios of the audio and video intercommunication system, and greatly meeting the audio and video intercommunication needs of users.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer data processing technology, specifically to audio and video processing technology, and more particularly to an audio and video intercommunication method, device, equipment and storage medium. Background Art

[0002] With the continuous development of technology, people's demand for security monitoring technology is also increasing. At present, relatively mature video monitoring systems have been implemented on the market. By binding the desktop client to one or more monitoring devices, the desktop client can be used to control each monitoring device for audio and video acquisition.

[0003] In the existing video surveillance system, different desktop clients can realize video on demand and video call functions. That is, one desktop client can demand the audio and video content collected by a monitoring device bound to another desktop client, or two desktop clients can realize audio and video calls.

[0004] When implemented, the existing video surveillance system can only transmit audio and video encoding files in a unified format collected by each desktop client. Furthermore, when the application scenario of video surveillance needs to be expanded, for example, when a new type of client for collecting audio and video encoding files in a new format is added to the video surveillance system, it is necessary to redesign the video surveillance system with a new architecture, and it is impossible to introduce the new type of client on the original video surveillance system. Summary of the invention

[0005] The embodiments of the present invention provide an audio and video intercommunication method, apparatus, device and storage medium to realize audio and video intercommunication between a web client and a desktop client in an existing audio and video intercommunication system based on a desktop client.

[0006] In a first aspect, an embodiment of the present invention provides an audio and video intercommunication method, wherein the method is executed by a server in an audio and video intercommunication system, and the audio and video streams in the audio and video intercommunication system are transmitted between different servers in a target coding format, and the method includes:

[0007] In response to an audio and video intercommunication request sent by a requested client in the same local area network, media negotiation is performed with the requested client;

[0008] After the media negotiation is passed, obtaining the requesting client that the requested client requests to communicate with, and detecting whether the requesting client is located in the same local area network as the requesting client;

[0009] If not, when the client type of the requested client is incompatible with the target coding format, converting the real-time audio and video stream collected by the requested client into an appropriate audio and video stream of the target coding format;

[0010] The dubbed video stream is forwarded to the next server that matches the requesting client.

[0011] In a second aspect, an embodiment of the present invention further provides an audio and video intercommunication device, wherein the device is executed by a server in an audio and video intercommunication system, and the audio and video streams in the audio and video intercommunication system are transmitted between different servers in a target coding format, and the device includes:

[0012] A media negotiation module, configured to respond to an audio and video intercommunication request sent by a requested client in the same local area network and perform media negotiation with the requested client;

[0013] A same LAN detection module, used for obtaining the requesting client that the requested client requests to communicate with after the media negotiation is passed, and detecting whether the requesting client is located in the same LAN as the requesting client;

[0014] The adaptive audio and video stream conversion module is used for converting the real-time audio and video stream collected by the requested client into an adaptive audio and video stream of the target coding format if it is detected that the client is not located in the same local area network as the requesting client and when the client type of the requested client is not compatible with the target coding format;

[0015] The dubbing-appropriate video stream forwarding module is used to forward the dubbing-appropriate video stream to the next server that matches the requesting client.

[0016] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the audio and video intercommunication method as described in any embodiment of the present invention is implemented.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the audio and video intercommunication method as described in any embodiment of the present invention is implemented.

[0018] The technical solution provided by the embodiment of the present invention responds to the audio and video intercommunication request sent by the requested client in the same local area network; after the media negotiation is passed, the requesting client that the requested client requests to intercommunication with is obtained, and whether it is located in the same local area network as the requesting client; if not, when the client type of the requested client is not compatible with the target coding format, the collected real-time audio and video stream is converted into an appropriate audio and video stream of the target coding format; and forwarded to the next server that matches the requesting client. Through the above method, the problem that the audio and video cannot be intercommunication caused by the inconsistent video coding format between the requested client and the requesting client is solved. Flexible audio and video interaction between the requested client and the requesting client can be achieved, the applicable scenarios of the audio and video intercommunication system are expanded, and the audio and video intercommunication needs of users are greatly met. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1a A flowchart of an audio and video intercommunication method provided in Embodiment 1 of the present invention;

[0020] Figure 1b A schematic diagram of a specific application scenario of the method provided in the first embodiment of the present invention;

[0021] Figure 2 A flowchart of another audio and video intercommunication method provided in Embodiment 2 of the present invention;

[0022] Figure 3 A flowchart of another audio and video intercommunication method provided in Embodiment 3 of the present invention;

[0023] Figure 4 It is a structural diagram of an audio and video intercommunication device provided by Embodiment 4 of the present invention;

[0024] Figure 5 It is a structural diagram of a computer device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0026] Embodiment 1

[0027] Figure 1aThis is a flow chart of an audio and video intercommunication method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where audio and video interaction is performed between a requested client and a requesting client. The method of this embodiment can be executed by an audio and video intercommunication device, which can be implemented by software and / or hardware, and the device can generally be configured in a server in an audio and video intercommunication system.

[0028] The audio and video intercommunication system is a communication system that has been built and is suitable for transmitting audio and video streams collected by desktop clients. The audio and video intercommunication system includes one or more servers, each of which can be directly connected to one or more desktop clients. When desktop clients located in different local area networks need to perform audio and video intercommunication, the audio and video streams to be transmitted need to be forwarded between different servers.

[0029] Accordingly, the method specifically comprises the following steps:

[0030] S110: In response to an audio and video intercommunication request sent by a requested client in the same local area network, perform media negotiation with the requested client.

[0031] Among them, the requested client can be understood as a client directly connected to the server, that is, the server can receive the audio and video streams directly sent by the requested client in the same local area network, wherein the client type of the requested client can be a web client or a desktop client.

[0032] Specifically, the web client can be understood as a method of accessing relevant web pages by simply entering a URL in a browser without installing a desktop client, that is, achieving functions similar to those of the desktop client.

[0033] The audio and video intercommunication request refers to a request sent by the requested client to send the audio and video stream collected by the requested client to other requesting clients for sharing. The audio and video stream collected by the requested client may refer to the audio and video content collected by the requested client through one or more monitoring devices (e.g., cameras) bound to it.

[0034] In a specific application scenario, when a requesting client in the audio and video intercommunication system places video on demand for one or more monitoring devices bound to the requested client, the requested client can first send an audio and video intercommunication request to the directly connected server after authorizing the video on demand demand, requesting that the collected audio and video streams be sent to the client requesting video on demand.

[0035] As mentioned above, the existing audio and video intercommunication system is implemented based on the desktop client. Furthermore, when each server in the audio and video intercommunication system constructs the entire signaling system, it is based on the data format of the audio and video stream collected by the desktop client. Furthermore, when the server needs to transmit the data format of the audio and video stream collected by the web client and adapted to the web client, it needs to first perform media negotiation with the web client.

[0036] The media negotiation refers to media negotiation based on SIP (Session Initiation Protocol) or SDP (Session Description Protocol) in WebRTC (Web Real-Time Communication) and being compatible with the existing negotiation mechanism.

[0037] S120: After the media negotiation is successful, obtain the requesting client that the requested client requests to communicate with, and detect whether the requesting client is located in the same local area network as the requesting client.

[0038] After the media negotiation is passed, it means that the requested client can obtain the requesting client that requests intercommunication. That is, the real-time audio and video stream collected by the requested client can be transmitted to the requesting client.

[0039] The requesting client may be a client that expects to obtain the audio and video streams collected by the requested client. Specifically, the client type of the requesting client may also be a desktop client or a web client.

[0040] In this embodiment, if it is determined that the server associated with the requested client and the requesting client are located in the same local area network, it means that the server can send the audio and video stream of the requested client to the requesting client by direct transmission. If it is determined that the server and the requesting client are located in different local area networks, it means that the server can send the audio and video stream of the requested client to the requesting client by forwarding.

[0041] It can be understood that, assuming that the requested client is a web client, when the requesting client is also a web type, it is first necessary to determine whether the server in the audio and video intercommunication system corresponding to the web client is in the same local area network as the requesting client. If they are in the same local area network, the audio and video stream can be sent directly from the web client to the requesting client.

[0042] Furthermore, if they are in different local area networks, it means that forwarding between servers is required before they can be transmitted to the requesting client. Therefore, the encoding format adapted by the audio and video stream on the web client needs to be converted into an encoding format adapted for transmission between servers, so that it can be transmitted to the server associated with the requesting client. When transmitting from the server to the requesting client, since the requesting client is a web page type, the audio and video stream needs to be converted into an encoding format before transmission.

[0043] In addition, when the requesting client is of desktop type, it is first necessary to determine whether the server in the audio and video intercommunication system corresponding to the web client is in the same local area network as the requesting client. If they are in the same local area network, the audio and video stream can be sent directly from the web client to the requesting client; however, due to the different type of the requesting client, the audio and video stream needs to be converted into an encoding format before transmission.

[0044] Furthermore, if they are in different local area networks, it means that forwarding between servers is required before they can be transmitted to the requesting client. Therefore, the encoding format adapted by the audio and video stream in the web client needs to be converted into an encoding format adapted for transmission between servers, so that it can be transmitted to the server associated with the requesting client. When transmitting from the server to the requesting client, since the requesting client is a desktop type (the audio and video stream encoding format transmitted between the desktop client and the server is the same), the audio and video stream can be directly transmitted from the server associated with the requesting client to the requesting client.

[0045] S130: If not, when the client type of the requested client is incompatible with the target coding format, convert the real-time audio and video stream collected by the requested client into an appropriate audio and video stream of the target coding format.

[0046] The target encoding format is the encoding format of the real-time audio and video stream collected by the desktop client.

[0047] Optionally, the client types include: desktop client and web client; wherein the desktop client matches the target encoding format.

[0048] The desktop client may be a desktop client software that the user needs to download and install first, through which the user can log in to the client interface, and in which the user can directly play audio and video on demand and make calls. The web client may be understood as a method of accessing the relevant web page by simply entering the URL in the browser without installing the desktop client, that is, realizing functions similar to the desktop client.

[0049] It is understandable that the encoding format used by the desktop client matches the target encoding format. The encoding format of the web client is inconsistent with the target encoding format, that is, the encoding format of the web client is inconsistent with the encoding format of the desktop client. In order for the web client and the desktop client to forward the audio and video stream, the audio and video stream needs to be converted to transmit the audio and video stream correctly.

[0050] Optionally, the target encoding format is an H.264 encoding format; and the media negotiation is a media negotiation based on a session initiation protocol or a session description protocol.

[0051] Among them, the H.264 encoding format is a new generation of digital video compression format after MPEG4 (Moving Pictures Experts Group) jointly proposed by the International Organization for Standardization and the International Telecommunication Union. H.264 is one of the video coding and decoding technology standards. H.264 is a digital video coding standard. The Session Initiation Protocol (SIP) is an IP (Internet Protocol) telephone signaling protocol proposed by the Internet Engineering Task Force. SIP is used to initiate sessions, can control the establishment and termination of multimedia sessions involving multiple participants, and can dynamically adjust and modify session attributes, such as session bandwidth requirements, transmitted media types (voice, video and data, etc.), media encoding and decoding formats, support for multicast and unicast, etc. The Session Description Protocol (SDP) is a session description format. SDP does not belong to the transmission protocol, but only uses different appropriate transmission protocols, including the Session Notification Protocol, the Session Initiation Protocol, the Real Time Streaming Protocol and the Hypertext Transfer Protocol.

[0052] Specifically, the target encoding format is H.264. It is understandable that the encoding format of the desktop client is also H.264. The encoding format of the web client is VP8. Specifically, VP8 is a video codec format that can provide higher quality video with less data and requires less processing power to play video, providing an ideal solution for Internet TV, interactive Internet TV and video conferencing companies that are committed to differentiating their products and services.

[0053] When the server and the requesting client are located in different local area networks, the real-time audio and video stream needs to be forwarded between different servers. Considering that the audio and video intercommunication system is an already implemented communication system for transmitting the audio and video streams collected by the desktop client, the underlying signaling protocols applicable to different servers all match the target encoding format. Accordingly, before forwarding the real-time audio and video stream, it is necessary to first convert it into an audio and video stream of the target encoding format, that is, into an audio and video stream with an appropriate dubbing.

[0054] S140: Forward the dubbed video stream to the next server that matches the requesting client.

[0055] Among them, the next server can be a server associated with the requesting client. Different requesting clients correspond to a lower-level server. When the audio and video streams to be sent are located on different servers (that is, different local area networks), they need to be forwarded to the relevant servers first, and then transmitted to the requesting client. When the audio and video streams need to be forwarded by multiple servers before they can be transmitted to the target requesting client, the video stream needs to be transmitted through multiple servers in the middle. Since it passes through multiple servers, the server is a cluster architecture.

[0056] It can be understood that if the requested client is a web client, the audio and video intercommunication request sent by the web client, assuming that the audio and video stream is to be transmitted to the requesting client B associated with the desktop client, the lower-level server corresponding to the requesting client B is server 1.

[0057] Specifically, since the audio and video intercommunication request is sent by the web client, the server corresponding to the web client is the server in the audio and video intercommunication system, and the requesting client that obtains the intercommunication request from the web client is the requesting client B.

[0058] First, it is necessary to determine whether the server in the audio and video intercommunication system and the requesting client B are in the same local area network. If not, it means that forwarding between servers is required before the audio and video stream can be forwarded to the requesting client B. Therefore, the audio and video stream is forwarded from the server in the audio and video intercommunication system to the next server, that is, server 1.

[0059] In addition, during the transmission of the audio and video streams, the encoding format needs to be converted to adapt to the transmission of the audio and video streams between the web client, the desktop client and the server.

[0060] Specifically, the server can determine the transmission path required to send the dubbed video stream to the requesting client based on the address information of the requesting client (e.g., IP address or MAC address), for example, forwarding from the local server to server A, forwarding from server A to server B, and directly sending from server B to the requesting client. Furthermore, server A can be determined to be the next server that matches the requesting client.

[0061] For example, assume that the requested client is a web client, and the web client is set as client A. Now it is necessary to send the audio and video stream to the requesting client, that is, client B. The target encoding format is H.264 encoding format, that is, the encoding format of the desktop client is H.264 encoding format, and the encoding format of the web client is VP8 encoding format. Specifically, the encoding format corresponding to client A is VP8 encoding format.

[0062] First, client A sends an audio and video intercommunication request to the server in the same local area network. The server in the audio and video intercommunication system parses the audio and video intercommunication request and performs media negotiation with client A through the SIP or SDP protocol based on WebRTC. When the media negotiation is successful, client B requested by client A for intercommunication can be obtained.

[0063] Furthermore, if it is determined that client A and client B are not located in the same local area network, the server in the audio and video intercommunication system converts the real-time audio and video stream in the VP8 encoding format collected by client A into an appropriate audio and video stream in the H.264 encoding format. Specifically, the H.264 encoding format is the encoding format of the real-time audio and video stream collected by desktop client A.

[0064] Correspondingly, the server in the audio and video intercommunication system will adapt the audio and video stream and forward it to the next server that matches client B.

[0065] The technical solution provided by the embodiment of the present invention responds to the audio and video intercommunication request sent by the requested client in the same local area network; after the media negotiation is passed, the requesting client that the requested client requests to intercommunication with is obtained, and whether it is located in the same local area network as the requesting client; if not, when the client type of the requested client is not compatible with the target coding format, the collected real-time audio and video stream is converted into an appropriate audio and video stream of the target coding format; and forwarded to the next server that matches the requesting client. Through the above method, the problem that the audio and video cannot be intercommunication caused by the inconsistent video coding format between the requested client and the requesting client is solved. Flexible audio and video interaction between the requested client and the requesting client can be achieved, the applicable scenarios of the audio and video intercommunication system are expanded, and the audio and video intercommunication needs of users are greatly met.

[0066] In an optional implementation of the present embodiment, after detecting and determining that the client is not located in the same local area network as the requesting client, it also includes: when the client type of the requested client is compatible with the target encoding format, forwarding the real-time audio and video stream collected by the requested client to the next server that matches the requesting client.

[0067] The client type may include a desktop type or a web page type. When the client type is a desktop type, it is determined that the requesting client is a desktop client; when the client type is a web page type, it is determined that the requesting client is a web page client. The desktop type may be a type in which the requesting client logs in through downloaded fixed software. The web page type may be a type in which the requesting client logs in on a web page corresponding to a URL by inputting a URL.

[0068] Specifically, if the requesting client and the server are not located in the same local area network, the server cannot directly send the required audio and video streams to the requesting client. Since the client type of the requested client is compatible with the target encoding format, it means that the requested client is a desktop client, so the real-time audio and video streams collected by the requested client can be forwarded to the next server that matches the requesting client.

[0069] It is understandable that when the audio and video stream is sent to the next target server, it is necessary to determine the type of the requesting client. If the requesting client is a desktop client, the audio and video stream can be directly forwarded to the requesting client. However, if the requesting client is a web client, the audio and video stream needs to be converted into a coding format before it can be sent to the requesting client.

[0070] For example, assume that the requested client is a desktop client, named client A, and now it is necessary to send audio and video streams to the requesting client, that is, client B. The target encoding format (that is, the encoding format of the desktop client) is the H.264 encoding format, and the encoding format of the web client is the VP8 encoding format. Specifically, the encoding format corresponding to client A is the H.264 encoding format.

[0071] First, when the client type of client A is compatible with the target encoding format, the real-time audio and video stream collected by client A is forwarded to the next server matched with client B;

[0072] Furthermore, it is necessary to determine the client type of client B. If client B is a desktop client, it means that the encoding format adopted by client B is the H.264 encoding format, so it can be directly forwarded to client B from the next server.

[0073] Correspondingly, if client B is a web client, it means that the encoding format used by client B is VP8 encoding format. Therefore, when forwarding from the next server to client B, the audio and video stream needs to be converted from H.264 encoding format to VP8 encoding format before it can be forwarded to client B.

[0074] The advantage of such a setting is that when the client type of the requested client is adapted to the target encoding format, the real-time audio and video stream collected by the requested client is forwarded to the next server that matches the requesting client. In this way, the audio and video stream interaction can be performed more accurately when the client type of the requested client is adapted to the target encoding format.

[0075] Specific application scenarios

[0076] like Figure 1b As shown, assuming that the requested client is a web client, named client A, it is now necessary to send audio and video streams to the requesting client, that is, client B. The target encoding format (that is, the encoding format of the desktop client) is the H.264 encoding format, and the encoding format of the web client is the VP8 encoding format. Specifically, the encoding format corresponding to client A is the VP8 encoding format. Assume that it is determined that client A and client B are not located in the same local area network. Client A corresponds to server A. If an audio and video stream is sent to client B, the audio and video stream needs to be forwarded to server B corresponding to client B first, and then forwarded to client B. Of course, assuming that client B is located on server N, multiple forwardings of the server are required to reach client B.

[0077] Embodiment 2

[0078] Figure 2 This is a flow chart of another audio and video intercommunication method provided by Embodiment 2 of the present invention. This embodiment is optimized based on the above embodiments, and in this embodiment, an implementation mode in which the server and the requesting client in the audio and video intercommunication system are located in the same local area network is added.

[0079] Accordingly, the method specifically comprises the following steps:

[0080] S210: In response to an audio and video intercommunication request sent by a requested client in the same local area network, perform media negotiation with the requested client.

[0081] S220: After the media negotiation is successful, obtain the requesting client that the requested client requests to communicate with.

[0082] S230, detecting whether the requesting client is located in the same local area network; if so, executing S240; if not, executing S270.

[0083] S240, detecting whether the client types of the requesting client and the requested client are consistent; if so, executing S250; if not, executing S260.

[0084] S250: Send the real-time audio and video stream directly to the requesting client.

[0085] S260. Convert the encoding format of the real-time audio and video stream according to the client type of the requesting client, and directly send the converted video stream to the requesting client.

[0086] In another optional implementation of this embodiment, when the server and the requesting client in the audio and video intercommunication system are located in the same local area network, it is necessary to detect whether the client types of the requesting client and the requested client are consistent. If they are consistent, that is, the client types of the requesting client and the requested client can both be desktop clients or web clients, so the real-time audio and video stream can be directly sent from the requested client to the requesting client.

[0087] Further, if they are inconsistent, that is, when the requested client is a desktop client, the requesting client is a web client; or when the requested client is a web client, the requesting client is a desktop client.

[0088] Specifically, when the requested client is a desktop client, the requesting client is a web client, and the encoding format of the real-time audio and video stream is converted according to the client type of the requesting client being a web client, and the converted video stream is directly sent to the requesting client. When the requested client is a web client, the requesting client is a desktop client, and the encoding format of the real-time audio and video stream is converted according to the client type of the requesting client being a desktop client, and the converted video stream is directly sent to the requesting client.

[0089] S270. When the client type of the requested client is incompatible with the target coding format, convert the real-time audio and video stream collected by the requested client into an appropriate audio and video stream of the target coding format.

[0090] S280: Forward the dubbed video stream to the next server that matches the requesting client.

[0091] In this embodiment, it is assumed that the requested client is a desktop client, and the requesting client that the requested client requests to communicate with is obtained and is set as the requesting client C.

[0092] First, check whether the client is in the same LAN as the requesting client C. If they are in the same LAN, it is necessary to obtain the client type of the requesting client C. When the requesting client C is a desktop client, it means that the client types of the requesting client C and the requested client are the same, and the audio and video stream can be sent directly from the desktop client to the requesting client C. Furthermore, when the requesting client C is a web client, it means that the client types of the requesting client C and the requested client are inconsistent, and it is necessary to convert the encoding format of the real-time audio and video stream according to the client type of the requesting client C being a web client, and send the converted video stream directly to the requesting client C.

[0093] Correspondingly, it is determined that the requesting client C is not located in the same local area network. Since the requested client is a desktop client and is adapted to the target encoding format, the real-time audio and video stream collected by the requested client is forwarded to the next server that matches the requesting client C.

[0094] In addition, assuming that the requested client is a web client, and it is determined that it is not located in the same local area network as the requesting client C, it can be known that the client type of the requested client is incompatible with the target encoding format. Therefore, it is necessary to convert the real-time audio and video stream captured by the requested client into an appropriate dubbing video stream in the target encoding format, and forward the appropriate dubbing video stream to the next server that matches the requesting client C.

[0095] The technical solution provided by the embodiment of the present invention performs media negotiation with the requested client by responding to the audio and video intercommunication request sent by the requested client in the same local area network; after the media negotiation is passed, obtains the requesting client that the requested client requests to intercommunicate with, and detects whether it is located in the same local area network as the requesting client; if so, detects whether the client types of the requesting client and the requested client are consistent; if so, sends the real-time audio and video stream directly to the requesting client; if not, converts the encoding format of the real-time audio and video stream according to the client type of the requesting client, and sends the converted video stream directly to the requesting client; if it is not located in the same local area network as the requesting client, then when the client type of the requested client is not compatible with the target encoding format, converts the real-time audio and video stream collected by the requested client into a dubbing video stream with an adapted dubbing format of the target encoding format; and forwards the adapted dubbing video stream to the next server that matches the requesting client. In this way, the interaction of audio and video streams between the requesting client and the requested client located in the same local area network can be determined more specifically, which expands the applicable scenarios of the existing audio and video stream intercommunication system and improves the user experience.

[0096] Embodiment 3

[0097] Figure 3This is a flow chart of another audio and video intercommunication method provided in Embodiment 3 of the present invention. This embodiment is optimized based on the above embodiments, and in this embodiment, a method for the server to process the forwarded audio and video streams forwarded by other servers is added.

[0098] Accordingly, the method specifically comprises the following steps:

[0099] S310. When receiving the transcribed audio video stream in the target coding format forwarded by other servers, obtain a target request client matching the transcribed audio video stream.

[0100] The transcribed audio and video stream may be a video stream sent by other servers received by a server in the audio and video intercommunication system. The target request client may be a request client to which the transcribed audio and video stream needs to be sent after being parsed by the server in the audio and video intercommunication system.

[0101] S320, detecting whether the target request client is located in the same local area network, if not, executing S330, and if so, executing S340.

[0102] S330: forward the transcribed audio video stream to the next server that matches the target requesting client.

[0103] S340: Obtain the client type of the target requesting client.

[0104] S350: If the client type of the target requesting client is compatible with the target encoding format, the transcribed audio video stream is directly sent to the target requesting client.

[0105] S360. If the client type of the target request client is not compatible with the target encoding format, the encoding format of the converted audio and video stream is converted according to the client type of the target request client, and the converted audio and video stream is directly sent to the target request client.

[0106] For example, assume that client E (which can be a desktop client or a web client) in the server of the audio and video intercommunication system receives a transcribed audio and video stream in H.264 encoding format, and through parsing, it is found that the target requesting client corresponding to the transcribed audio and video stream is client F.

[0107] Furthermore, the server in the audio and video intercommunication system needs to detect whether it is in the same LAN as the client F. If it is not in the same LAN as the client F, the server in the audio and video intercommunication system will transfer the audio and video stream to the next server matching the client F.

[0108] Correspondingly, if the client F is located in the same local area network as the client F, the client type of the client F is obtained. If it is determined that the client type of the client F is a desktop type, it is necessary to determine whether the client type of the client F is compatible with the target encoding format. It can be known that if the client type of the client F is compatible with the target encoding format, the transcribed audio and video stream will be sent directly to the client F. If it is determined that the client type of the client F is a web page type, it can be known that the client type of the client F is not compatible with the target encoding format, then it is necessary to convert the encoding format of the transcribed audio and video stream according to the client type of the client F, that is, convert the H.264 encoding format into the VP8 encoding format, and the converted audio and video stream is sent directly to the client F.

[0109] The technical solution provided by the embodiment of the present invention is to obtain the target request client that matches the transcribed audio and video stream when receiving the transcribed audio and video stream of the target coding format forwarded by other servers, and detect whether it is located in the same local area network as the target request client; if so, obtain the client type of the target request client; if the client type of the target request client is compatible with the target coding format, send the transcribed audio and video stream directly to the target request client; if the client type of the target request client is not compatible with the target coding format, convert the coding format of the transcribed audio and video stream according to the client type of the target request client, and send the converted audio and video stream directly to the target request client. In this way, the processing method of receiving the transcribed audio and video stream forwarded by other servers can be determined, and the received transcribed audio and video stream can be resent, and multiple forwarding of audio and video streams can be realized between the requested client and the requesting client, so that the audio and video interaction between the requested client and the requesting client can be realized in a more diversified way, and the applicable scenarios of the existing audio and video intercommunication system are expanded.

[0110] Embodiment 4

[0111] Figure 4 is a structural diagram of an audio and video intercommunication device provided in Embodiment 4 of the present invention. The audio and video intercommunication device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a server to implement an audio and video intercommunication method in an embodiment of the present invention. Figure 4 As shown, the device may specifically include: a media negotiation module 410 , a same LAN detection module 420 , a dubbing-appropriate video stream conversion module 430 , and a dubbing-appropriate video stream forwarding module 440 .

[0112] The media negotiation module 410 is used to respond to the audio and video intercommunication request sent by the requested client in the same local area network and perform media negotiation with the requested client;

[0113] The same LAN detection module 420 is used to obtain the requesting client that the requested client requests to communicate with after the media negotiation is passed, and detect whether the requesting client is located in the same LAN as the requesting client;

[0114] The audio / video stream conversion module 430 is configured to convert the real-time audio / video stream collected by the requested client into an audio / video stream of the target coding format if it is detected that the requested client is not located in the same local area network as the requesting client and when the client type of the requested client is not compatible with the target coding format;

[0115] The dubbing-appropriate video stream forwarding module 440 is used to forward the dubbing-appropriate video stream to the next server that matches the requesting client.

[0116] The technical solution provided by the embodiment of the present invention responds to the audio and video intercommunication request sent by the requested client in the same local area network; after the media negotiation is passed, the requesting client that the requested client requests to intercommunication with is obtained, and whether it is located in the same local area network as the requesting client; if not, when the client type of the requested client is not compatible with the target coding format, the collected real-time audio and video stream is converted into an appropriate audio and video stream of the target coding format; and forwarded to the next server that matches the requesting client. Through the above method, the problem that the audio and video cannot be intercommunication caused by the inconsistent video coding format between the requested client and the requesting client is solved. Flexible audio and video interaction between the requested client and the requesting client can be achieved, the applicable scenarios of the audio and video intercommunication system are expanded, and the audio and video intercommunication needs of users are greatly met.

[0117] On the basis of the above embodiments, it also includes a client type detection module, which can be specifically used to: after detecting whether the requesting client is located in the same local area network, if so, detect whether the client types of the requesting client and the requested client are consistent; if so, send the real-time audio and video stream directly to the requesting client; if not, convert the encoding format of the real-time audio and video stream according to the client type of the requesting client, and send the converted video stream directly to the requesting client.

[0118] On the basis of the above embodiments, it also includes an audio and video stream forwarding module, which can be specifically used to: after detecting and determining that the requested client is not located in the same local area network as the requesting client, when the client type of the requested client is compatible with the target encoding format, forward the real-time audio and video stream collected by the requested client to the next server that matches the requesting client.

[0119] On the basis of the above embodiments, it also includes a request client acquisition unit, which can be specifically used to: when receiving the transcribed audio video stream in the target coding format forwarded by other servers, obtain the target request client that matches the transcribed audio video stream, and detect whether it is located in the same local area network as the target request client; if so, obtain the client type of the target request client; if the client type of the target request client is compatible with the target coding format, send the transcribed audio video stream directly to the target request client; if the client type of the target request client is not compatible with the target coding format, convert the coding format of the transcribed audio video stream according to the client type of the target request client, and send the converted audio and video stream directly to the target request client.

[0120] On the basis of the above embodiments, it also includes a transcribed audio video stream forwarding module, which can be specifically used to: if it is detected that the transcribed audio video stream is not located in the same local area network as the target request client, forward the transcribed audio video stream to the next server that matches the target request client.

[0121] On the basis of the above embodiments, it can also be used that the client type includes: a desktop client and a web client; wherein the desktop client matches the target encoding format.

[0122] On the basis of the above embodiments, it can be specifically used that: the target encoding format is the H.264 encoding format; and the media negotiation is the media negotiation based on the Session Initiation Protocol or the Session Description Protocol.

[0123] The above-mentioned audio and video intercommunication device can execute the audio and video intercommunication method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0124] Embodiment 5

[0125] Figure 5 Schematic diagram of the structure of a computer device provided by Embodiment 5 of the present invention. Figure 5 As shown, the device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of processors 510 in the device can be one or more. Figure 5 A processor 510 is taken as an example; the processor 510, the memory 520, the input device 530 and the output device 540 in the device can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0126] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the audio and video intercommunication method in the embodiment of the present invention (for example, the media negotiation module 410, the same local area network detection module 420, the appropriate dubbing video stream conversion module 430 and the appropriate dubbing video stream forwarding module 440). The processor 510 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 520, that is, to implement the above-mentioned audio and video intercommunication method, which includes: responding to the audio and video intercommunication request sent by the requested client in the same local area network, media negotiation with the requested client; after the media negotiation is passed, obtaining the requesting client requested by the requested client for intercommunication, and detecting whether it is located in the same local area network as the requesting client; if not, when the client type of the requested client is not compatible with the target coding format, converting the real-time audio and video stream collected by the requested client into an appropriate dubbing video stream of the target coding format; forwarding the appropriate dubbing video stream to the next server matching the requesting client.

[0127] The memory 520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The input device 530 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 540 may include a display device such as a display screen.

[0129] Embodiment 6

[0130] Embodiment 6 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute an audio and video intercommunication method when executed by a computer processor, the method comprising: in response to an audio and video intercommunication request sent by a requested client in the same local area network, performing media negotiation with the requested client; after the media negotiation is passed, obtaining the requesting client that the requested client requests to intercommunicate with, and detecting whether it is located in the same local area network as the requesting client; if not, when the client type of the requested client is not compatible with the target encoding format, converting the real-time audio and video stream collected by the requested client into an adapted dubbing video stream in the target encoding format; and forwarding the adapted dubbing video stream to the next server that matches the requesting client.

[0131] Of course, the computer-readable storage medium provided in the embodiment of the present invention includes computer-readable instructions that are not limited to the method operations described above, and can also execute related operations in the audio and video intercommunication method provided in any embodiment of the present invention.

[0132] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0133] It is worth noting that in the embodiment of the above-mentioned audio and video intercommunication device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0134] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for audio and video intercommunication, It is characterized in that The method is executed by a server in an audio and video intercommunication system, wherein the audio and video streams in the audio and video intercommunication system are transmitted between different servers in a target coding format, and the method includes: In response to an audio and video intercommunication request sent by a requested client in the same local area network, media negotiation is performed with the requested client; After the media negotiation is passed, obtaining the requesting client that the requested client requests to communicate with, and detecting whether the requesting client is located in the same local area network as the requesting client; If not, when the client type of the requested client is incompatible with the target coding format, converting the real-time audio and video stream collected by the requested client into an appropriate audio and video stream of the target coding format; Forwarding the dubbed video stream to the next server that matches the requesting client; Among them, after detecting whether the requesting client is located in the same local area network, it also includes: if so, detecting whether the client types of the requesting client and the requested client are consistent; if so, sending the real-time audio and video stream directly to the requesting client; if not, converting the encoding format of the real-time audio and video stream according to the client type of the requesting client, and sending the converted video stream directly to the requesting client.

2. The method according to claim 1, It is characterized in that After detecting and determining that the client is not located in the same local area network as the requesting client, the method further includes: When the client type of the requested client is compatible with the target encoding format, the real-time audio and video streams collected by the requested client are forwarded to the next server that matches the requesting client.

3. The method according to claim 1, It is characterized in that Also includes: When receiving the transcribed audio video stream in the target coding format forwarded by other servers, obtaining a target request client matching the transcribed audio video stream, and detecting whether the target request client is located in the same local area network; If yes, obtaining the client type of the target requesting client; If the client type of the target requesting client is compatible with the target encoding format, the transcribed audio video stream is directly sent to the target requesting client; If the client type of the target request client is not compatible with the target encoding format, the encoding format of the converted audio and video stream is converted according to the client type of the target request client, and the converted audio and video stream is directly sent to the target request client.

4. The method according to claim 3, It is characterized in that After detecting whether the target request client is located in the same local area network, the method further includes: If not, the transcribed audio video stream is forwarded to the next server that matches the target request client.

5. The method according to any one of claims 1 to 4, It is characterized in that The client types include: desktop client and web client; Wherein, the desktop client matches the target encoding format.

6. The method according to any one of claims 1 to 4, It is characterized in that The target encoding format is the H.264 encoding format; The media negotiation is a media negotiation based on a session initiation protocol or a session description protocol.

7. An audio and video intercommunication device, It is characterized in that The device is executed by a server in an audio and video intercommunication system, wherein the audio and video streams in the audio and video intercommunication system are transmitted between different servers in a target coding format, and the device comprises: A media negotiation module, configured to respond to an audio and video intercommunication request sent by a requested client in the same local area network and perform media negotiation with the requested client; A same LAN detection module, used for obtaining the requesting client that the requested client requests to communicate with after the media negotiation is passed, and detecting whether the requesting client is located in the same LAN as the requesting client; The adaptive audio and video stream conversion module is used for converting the real-time audio and video stream collected by the requested client into an adaptive audio and video stream of the target coding format if it is detected that the client is not located in the same local area network as the requesting client and when the client type of the requested client is not compatible with the target coding format; A dubbing-appropriate video stream forwarding module, used for forwarding the dubbing-appropriate video stream to a next server matching the requesting client; The client type detection module is used to: after detecting whether the requesting client is located in the same local area network, if so, detect whether the client types of the requesting client and the requested client are consistent; if so, send the real-time audio and video stream directly to the requesting client; if not, convert the encoding format of the real-time audio and video stream according to the client type of the requesting client, and send the converted video stream directly to the requesting client.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the audio and video intercommunication method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the audio and video intercommunication method according to any one of claims 1 to 6 is implemented.

Images