Audio and video communication system

Abstract

The invention discloses an audio and video communication system, and belongs to the technical field of communication. Comprising a signaling receiving and transmitting module used for receiving and transmitting call signaling; the main thread module is connected with the signaling transceiving module and is used for outputting an audio type identification result and a video type identification result according to the type of the call signaling; the audio processing module is connected with the main thread module, and decodes or encodes the received audio file based on the audio type identification result; and the video processing module is connected with the main thread module and is used for monitoring and outputting the received video file based on the video type identification result. The beneficial effects of the technical scheme are that the method is designed for the client, realizes signaling transmission and packaging and sending of audio and video streams, and has good integrity and adaptability.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a client communication system. Background Art

[0002] The full name of GB / T28181 is "GB / T 28181-2016 Technical Requirements for Information Transmission, Exchange and Control of Public Security Video Surveillance Networking Systems". At present, the patent applications based on GB / T28181 have been published, such as: the Chinese invention patent application with publication number CN115103025A discloses a PTZ device control method based on the GBT28181 protocol, and this patent application mainly provides an implementation solution for PTZ control. The Chinese invention patent application with publication number CN111757044A discloses a GB / T28181-2016 signaling server architecture method based on OSIP, and this patent application only provides an implementation solution for the server architecture. The Chinese invention patent application with publication number CN111193801A discloses a GB 28181 platform distributed cluster implementation system based on JAIN-SIP, which is also designed from the perspective of the server. As for the implementation solution for the client, some open source solutions currently lack integrity, usually staying at the level of SIP signaling reception and transmission, and lack the function of packaging and sending audio and video streams. Summary of the invention

[0003] The purpose of the present invention is to provide an audio and video communication system to solve the above technical problems;

[0004] An audio and video communication system, comprising:

[0005] Signaling transceiver module, used for receiving and sending call signaling;

[0006] A main thread module, connected to the signaling transceiver module, configured to output an audio type recognition result and a video type recognition result according to the type of the call signaling;

[0007] An audio processing module, connected to the main thread module, and decoding or encoding the received audio file based on the audio type recognition result;

[0008] The video processing module is connected to the main thread module and monitors and outputs the received video file based on the video type identification result.

[0009] Preferably, the signaling transceiver module includes:

[0010] A signaling receiving unit, configured to receive the call signaling;

[0011] A signaling parsing unit, connected to the signaling receiving unit, configured to extract a characteristic field in the call signaling and parse the type of the call signaling;

[0012] A signaling sending unit, connected to the signaling parsing unit, is used to send the type of the call signaling to the main thread module.

[0013] Preferably, the audio processing module includes:

[0014] An audio decoding unit, wherein the input end of the audio decoding unit is connected to the server, and the output end of the audio decoding unit is connected to the mixing module, for receiving the encoded audio sent by the server and decoding it into the original audio according to a preset decoding strategy.

[0015] Preferably, the audio processing module further includes:

[0016] An audio encoding module, wherein the input end of the audio encoding module is connected to the mixing module, and the output end of the audio encoding module is connected to the server, and is used to encode the original audio according to a preset encoding strategy and then transmit it to the server.

[0017] Preferably, the preset decoding strategy is to decode and quantize the encoded audio according to G.711 encoding to reconstruct the original audio.

[0018] Preferably, the preset encoding strategy is to quantize the original audio, map it to the value range of G.711 encoding, convert the quantized value into binary code, and obtain the encoded audio.

[0019] Preferably, the video processing module includes:

[0020] A video monitoring unit, the input end of which is connected to the audio and video module and the server respectively, and is used to control the resolution and bit rate of the video stream sent by the audio and video module according to the report packet sent by the server;

[0021] A video stream sending unit, wherein the output end of the video stream sending unit is respectively connected to the video monitoring unit and the server, and is used to send the video stream to the server.

[0022] Preferably, the report packet includes the total number of video data packets, the number of bytes, and the packet loss rate received by the server.

[0023] Preferably, the main thread module includes:

[0024] A configuration unit, connected to the audio and video configuration module, for dynamically configuring the audio and video configuration module according to a configuration file;

[0025] A trigger unit is connected to the signaling transceiver module and is used to trigger the audio processing module and the video processing module according to the type of the call signaling.

[0026] Preferably, the call signaling is transmitted between the server and the signaling transceiver module via a protocol stack;

[0027] The protocol stack includes Osip2 and eXosip2.

[0028] The beneficial effects of the present invention are: it is designed for the client, realizes signaling transmission, and packing and sending of audio and video streams, and has good integrity and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of an audio and video communication system of the present invention;

[0030] Figure 2 is a schematic diagram of a signaling transceiver module of the present invention;

[0031] Figure 3 It is a schematic diagram of the main thread module of the present invention.

[0032] In the attached figure: 1. Signaling transceiver module; 11. Signaling receiving unit; 12. Signaling parsing unit; 13. Signaling sending unit; 2. Main thread module; 21. Configuration unit; 22. Trigger unit; 3. Audio processing module; 31. Audio decoding unit; 32. Audio encoding module; 4. Video processing module; 41. Video monitoring unit; 42. Video stream sending unit; 5. Audio and video configuration module; 6. Mixing module; 7. Audio and video module; 8. Protocol stack; 9. Server; 10. Configuration file. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0035] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0036] An audio and video communication system, such as Figure 1 As shown, including

[0037] Signaling transceiver module 1, used for receiving and sending call signaling;

[0038] The main thread module 2 is connected to the signaling transceiver module 1 and is used to output the audio type recognition result and the video type recognition result according to the type of the call signaling;

[0039] The audio processing module 3 is connected to the main thread module 2 and decodes or encodes the received audio file based on the audio type recognition result;

[0040] The video processing module 4 is connected to the main thread module 2, and monitors and outputs the received video files based on the video type recognition result.

[0041] Specifically, the present invention provides an audio and video communication system for client communication, which performs signaling transmission through a signaling transceiver module 1, triggers an audio processing module 3 and a video processing module 4 through a main thread module 2, implements audio and video stream packaging and transmission based on the national standard GB / T28181, and can quickly adapt to various GB / T28181 servers 9. Among them, the call signaling is SIP (Session Initiation Protocol) signaling.

[0042] In a preferred embodiment, referring to Figure 2 , the signaling transceiver module 1 includes,

[0043] A signaling receiving unit 11, configured to receive a call signaling;

[0044] The signaling parsing unit 12 is connected to the signaling receiving unit 11 and is used to extract the characteristic field in the call signaling and parse the type of the call signaling;

[0045] The signaling sending unit 13 is connected to the signaling parsing unit 12 and is used to send the type of the call signaling to the main thread module 2.

[0046] Specifically, after the signaling transceiver module 1 is started, it continuously monitors the SIP signaling receiving port set in the configuration file 10, and sends a registration message to the peer server 9. After the registration action is successfully completed, the heartbeat keep-alive signaling is periodically sent to the peer server 9 according to the configuration file 10, and the SIP signaling sent by the server 9 is continuously received and responded.

[0047] The types of call signaling include media stream request messages and audio intercom request messages.

[0048] Reference Figure 1 In the direction of the green arrow, when the signaling transceiver module 1 receives the media stream request message sent by the peer server 9, it will parse the message and parse the category of the media stream requested by the server 9: audio stream, video stream, or audio and video stream. Then, according to the agreement of the GB / T28181 standard, the signaling interaction process is completed. The audio processing module 3 and the video processing module 4 are started according to the specific stream category.

[0049] In a preferred embodiment, the audio processing module 3 includes:

[0050] The audio decoding unit 31 refers to Figure 1 In the direction of the blue arrow, the input end of the audio decoding unit 31 is connected to the server 9, and the output end of the audio decoding unit 31 is connected to the mixing module 6, which is used to receive the encoded audio sent by the server 9 and decode it into the original audio according to the preset decoding strategy;

[0051] The audio processing module 3 also includes,

[0052] Audio encoding module 32, refer to Figure 1 As shown by the blue arrow, the input end of the audio encoding module 32 is connected to the mixing module 6, and the output end of the audio encoding module 32 is connected to the server 9, which is used to encode the original audio according to the preset encoding strategy and then transmit it to the server 9.

[0053] Specifically, the audio encoding module 32 receives the original audio data (PCM) from the external mixing module 6, and then encodes the PCM data into G.711 audio encoding data through the built-in software encoding program, loads the RTP message header according to the GB / T28181 standard, and sends it to the peer server 9 through the UDP protocol.

[0054] The audio and video module 7 and the audio mixing module 6 are external modules outside the present invention, and the audio and video module 7 and the audio mixing module 6 involve PCM (Pulse Code Modulation), RTP (Real-time Transport Protocol) and UDP (User Datagram Protocol).

[0055] PCM is a way of representing analog signals as digital signals, by sampling and quantizing the analog signal to convert it into a digital format.

[0056] RTP is a protocol for transmitting audio and video over IP networks for real-time applications such as VoIP and video conferencing.

[0057] UDP is a simple transport layer protocol used to send datagrams across a network.

[0058] When the signaling transceiver module 1 receives the audio intercom request message sent by the peer server 9, it will complete the signaling interaction process of the audio intercom according to the GB / T28181 standard, and then start the audio encoding module 32 and the audio decoding unit 31.

[0059] The audio decoding unit 31 receives the G.711 encoded audio data sent by the peer server 9, decodes the G.711 audio data into PCM original audio data through the built-in software audio decoding program, and then sends it to the external audio and video module 7 for playback. This process is synchronized with the operation of the audio encoding module 32, thereby realizing the audio intercom function.

[0060] In a preferred embodiment, the preset decoding strategy is to decode the encoded audio according to G.711 encoding and then quantize it to reconstruct the original audio.

[0061] Specifically, the G.711-encoded audio data stream is obtained from the peer server 9. Usually, this is transmitted over the network or read from local storage. Then, the encoding type is determined: A-law or μ-law (mu-law).

[0062] Using the A-law decoding algorithm:

[0063] G.711 A-law encoding compresses samples into 8-bit values, extracts the 8-bit encoded bytes, and maps the bytes back to their linear PCM values ​​according to the A-law decoding table. Finally, the linear PCM values ​​are converted to the corresponding sample values.

[0064] Using the μ-law decoding algorithm:

[0065] Extract the 8-bit encoded bytes, perform reverse mapping according to the μ-law decoding table, and convert the mapped values ​​into PCM samples. Map the decoded PCM values ​​to the target audio format (e.g. 16-bit PCM). Ensure that the number of bits of the audio samples matches the playback device. Send the quantized PCM data to the audio playback device or further processing module to complete the audio reconstruction and playback.

[0066] The G.711 encoded audio data received by the audio decoding unit 31 is decoded and quantized according to the G.711 encoding method (such as A-law or μ-law) to obtain PCM original audio data.

[0067] In a preferred embodiment, the preset encoding strategy is to quantize the original audio, map it to the value range of G.711 encoding, convert the quantized value into binary code, and obtain the encoded audio.

[0068] Specifically, the audio encoding module 32 quantizes the PCM data and maps the quantized PCM values ​​to the value range of the G.711 code, that is, converts the PCM values ​​into corresponding G.711 code values ​​(for example, converts the PCM sample values ​​into their representation under A-law or μ-law coding). Then, the quantized and mapped G.711 code values ​​are converted into binary form for subsequent transmission.

[0069] After loading the RTP header according to the GB / T28181 standard, it is sent to the peer server 9 via the UDP protocol. Before sending the encoded audio data, it is necessary to construct an RTP (Real-time Transport Protocol) header. The RTP header contains a version number, a sequence number, a timestamp, and a load type. The encoded audio data and the constructed RTP header are combined into a complete RTP packet for network transmission.

[0070] The RTP packet is encapsulated by UDP (User Datagram Protocol), which provides a connectionless transmission service suitable for the transmission of real-time audio streams. The encapsulated UDP packet is sent to the target server 9 through the network. The target address and port need to be configured before sending.

[0071] In a preferred embodiment, the video processing module 4 includes:

[0072] Video surveillance unit 41, see Figure 1 In the direction of the black arrow, the input end of the video monitoring unit 41 is connected to the audio and video module 7 and the server 9 respectively, and is used to control the resolution and bit rate of the video stream sent by the audio and video module 7 according to the report packet sent by the server 9;

[0073] The video stream sending unit 42 refers to Figure 1 In the direction of the black arrow, the output end of the video stream sending unit 42 is connected to the video monitoring unit 41 and the server 9 respectively, for sending the video stream to the server 9;

[0074] The report packet includes the total number of video data packets, the number of bytes, and the packet loss rate received by the server 9 .

[0075] Specifically, when the video stream sending unit 42 is started, the video monitoring unit 41 will also be started. The video stream sending unit 42 will receive the encoded H.264 video data from the external audio and video module 7, and then load the RTP message header according to the GB / T28181 standard, and package it into a PS video stream, and then send it to the peer server 9 via the UDP protocol.

[0076] The video monitoring unit 41 is used to monitor the video stream transmission status, control the video transmission resolution and bit rate in real time, and ensure the smoothness and integrity of the video stream when the network environment is not good. The video monitoring unit 41 will try to receive the RTCP receiver report from the peer server 9. The report contains the total number of video data packets received by the server 9, the specific number of bytes, and the packet loss rate. The video monitoring unit 41 controls the bit rate and resolution of the video transmission in real time according to the packet loss rate to ensure that the server can play the video stream smoothly. It has a video flow control function and is highly practical.

[0077] The audio and video module 7 and the video processing module 4 involve H.264 (an encoding format for transmitting video data), RTP (a protocol for real-time transmission of video data, which packages the H.264-encoded video stream into data packets that can be transmitted over the network), and UDP.

[0078] In a preferred embodiment, referring to Figure 3 , the main thread module 2 includes,

[0079] Configuration unit 21, reference Figure 1 In the direction of the orange arrow, the configuration unit 21 is connected to the audio and video configuration module 5, and is used to dynamically configure the audio and video configuration module 5 according to the configuration file 10;

[0080] The trigger unit 22 is connected to the signaling transceiver module 1 and is used to trigger the audio processing module 3 and the video processing module 4 according to the type of the call signaling.

[0081] Specifically, when the main thread module 2 is started, it will read the configuration file 10 and pre-configure each module. It can also be configured according to the user's settings during operation.

[0082] According to the specific configuration, the signaling transceiver module 1 is first started, and then the audio processing module 3 and the video processing module 4 are started as needed according to the received SIP signaling. The configuration file 10 is monitored in real time, and the configuration changes are synchronously sent to the external audio and video module 7 through the audio and video configuration module 5.

[0083] In a preferred embodiment, the server 9 and the signaling transceiver module 1 transmit call signaling via the protocol stack 8;

[0084] The protocol stack 8 includes Osip2 and eXosip2.

[0085] Specifically, Osip2 is a lightweight library for the SIP protocol. It provides basic SIP functions, including request and response processing, session management, etc. eXosip2 is an extended library based on Osip2, including support for multimedia sessions, such as audio and video stream management, making the establishment and maintenance of SIP connections simpler and more efficient.

[0086] The present invention includes client initialization configuration, signaling transmission, audio and video stream packaging and sending, and flow control, has good integrity and adaptability, and can quickly adapt to various GB / T28181 servers 9. Signaling interaction is completed in accordance with the provisions of the GB / T28181 standard, and audio and video stream packaging and sending are implemented in accordance with the recommended scheme of the GB / T28181 standard. It can adapt to the functional interaction of most servers 9 of the GB / T28181 signaling specification, and has strong versatility. The technical implementation has a clear hierarchical structure, and the expansion and secondary development can be completed by modifying the interface of the audio and video module 7 and the interface of the configuration file 10.

[0087] The above description is only a preferred embodiment of the present invention, and does not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. An audio and video communication system, characterized in that: include, Signaling transceiver module, used for receiving and sending call signaling; A main thread module, connected to the signaling transceiver module, configured to output an audio type recognition result and a video type recognition result according to the type of the call signaling; An audio processing module, connected to the main thread module, and decoding or encoding the received audio file based on the audio type recognition result; The video processing module is connected to the main thread module and monitors and outputs the received video file based on the video type identification result.

2. The audio and video communication system according to claim 1, characterized in that: The signaling transceiver module includes: A signaling receiving unit, configured to receive the call signaling; A signaling parsing unit, connected to the signaling receiving unit, configured to extract a characteristic field in the call signaling and parse the type of the call signaling; A signaling sending unit, connected to the signaling parsing unit, is used to send the type of the call signaling to the main thread module.

3. The audio and video communication system according to claim 1, characterized in that: The audio processing module includes: An audio decoding unit, wherein the input end of the audio decoding unit is connected to the server, and the output end of the audio decoding unit is connected to the mixing module, for receiving the encoded audio sent by the server and decoding it into the original audio according to a preset decoding strategy.

4. The audio and video communication system according to claim 3, characterized in that: The audio processing module also includes: An audio encoding module, wherein the input end of the audio encoding module is connected to the mixing module, and the output end of the audio encoding module is connected to the server, and is used to encode the original audio according to a preset encoding strategy and then transmit it to the server.

5. The audio and video communication system according to claim 3, characterized in that: The preset decoding strategy is to decode and quantize the encoded audio according to G.711 encoding to reconstruct the original audio.

6. The audio and video communication system according to claim 4, characterized in that: The preset encoding strategy is to quantize the original audio, map it to the value range of G.711 encoding, convert the quantized value into binary code, and obtain the encoded audio.

7. The audio and video communication system according to claim 3, characterized in that: The video processing module includes: A video monitoring unit, the input end of which is connected to the audio and video module and the server respectively, and is used to control the resolution and bit rate of the video stream sent by the audio and video module according to the report packet sent by the server; A video stream sending unit, wherein the output end of the video stream sending unit is respectively connected to the video monitoring unit and the server, and is used to send the video stream to the server.

8. The audio and video communication system according to claim 7, characterized in that: The report packet includes the total number of video data packets, the number of bytes, and the packet loss rate received by the server.

9. The audio and video communication system according to claim 1, characterized in that: The main thread module includes: A configuration unit, connected to the audio and video configuration module, for dynamically configuring the audio and video configuration module according to a configuration file; A trigger unit is connected to the signaling transceiver module and is used to trigger the audio processing module and the video processing module according to the type of the call signaling.

10. The audio and video communication system according to claim 3, characterized in that: The server and the signaling transceiver module transmit the call signaling via a protocol stack; The protocol stack includes Osip2 and eXosip2.

Images