An audio and video online communication system and method based on combination of 5G message and WebRTC

Abstract

The application discloses an audio and video online communication system and method based on combination of 5G message and WebRTC, and the audio and video online alarm system comprises a 5G user terminal, an interrogator server, a signaling server, a logic processing server, a media server, a call plan database and a video storage database; the 5G user terminal is connected with the signaling server, the media server and the logic processing server respectively; the interrogator server is connected with the signaling server, the media server and the logic processing server respectively; the signaling server is connected with the logic processing server and the media server respectively; the logic processing server is connected with the call plan database; and the media server is connected with the video database. The application does not need to download and install an application and can communicate through a video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 5G message, WebRTC, audio and video, and more particularly to an audio and video online communication system and method based on the combination of 5G message and WebRTC. BACKGROUND

[0002] In specific business scenarios such as public safety response, emergency service dispatch, etc., there is often a need for timely, reliable and information-rich remote communication between the information initiator (i.e. the inquiring personnel) and the professional responder (i.e. the inquired personnel). The traditional communication method mainly relies on voice telephone or direct interview. However, the voice telephone communication method has significant defects: first, it is not intuitive enough, when the information initiator cannot accurately describe the on-site situation due to nervousness, knowledge level limitation or complex environment, it will seriously affect the communication efficiency and information accuracy; second, for the hearing-impaired or speech-impaired (such as the deaf-mute), voice telephone is completely unusable, making it difficult for them to seek help effectively in an emergency; third, the professional responder cannot obtain the facial expression, body language or environmental visual information of the other party through the telephone, which may miss key non-verbal clues and increase the difficulty of identity verification and situation determination, thus possibly delaying the critical decision-making time.

[0003] In addition, some existing Internet-based video call solutions often require users to pre-download and install specific application programs (Apps), which constitutes a usage threshold in emergency situations, delaying the golden time of communication. At the same time, these general solutions cannot meet the stringent requirements for safety and credibility in the above-mentioned specific scenarios in terms of communication reliability and connection enforceability. SUMMARY

[0004] The present application provides an audio and video online communication system and method based on the combination of 5G message and WebRTC, which enables users to communicate through video without the need to download and install applications, making it more intuitive and time-saving.

[0005] To solve the above problems, an audio and video online communication system based on the combination of 5G message and WebRTC is provided, comprising:

[0006] 5G user terminal, inquiring personnel server, signaling server, logic processing server, media server, call plan database, video storage database; the 5G user terminal is connected to the signaling server, media server and logic processing server respectively; the inquiring personnel server is connected to the signaling server, media server and logic processing server respectively; the signaling server is connected to the logic processing server and media server respectively; the logic processing server is connected to the call plan database; the media server is connected to the video database;

[0007] The 5G user terminal belongs to an interrogator and is used to subscribe to 5G messages and has a camera and microphone function and Internet access capability.

[0008] The interrogator server belongs to an online interrogatee and has a camera and microphone function and Internet access capability; and can maintain an online state by continuously initiating a registration request to a signaling server or a media server.

[0009] The logic processing server can assign interrogated persons for interrogative audio and video calls, and generate a call plan, which includes interrogator information and interrogator server information; is used to maintain a heartbeat connection with the interrogator server and update location information in a call plan database when the connection is disconnected; can interface with the 5G user terminal and the interrogator server, perform identity authentication and registration information, interface with the call plan database, generate a call relationship, store call plan data when the call relationship is closed, interface with the media server, select the best media route through a communication acceleration capability method and establish routing information, interface with the signaling server, query call plan information for the signaling server, and return to a call object; and can generate a call information after the audio and video call ends, which includes audio and video call information and call information and media resource storage address.

[0010] The call plan database is used to store call information, so that the signaling server can query call information through the call plan database, and the logic processing server can add, delete, modify and query call plans through the call plan database; and store call information.

[0011] The signaling server can interface with the logic processing server, establish a connection with the 5G user terminal when calling and obtain routing information of the interrogator server from the logic processing server and establish a connection therewith.

[0012] The video database is used to objectify and encrypt audio and video calls to store audio and video storage data; the audio and video storage data correspond one-to-one to call plan data; and the logic processing server can add, delete, modify and query the audio and video database.

[0013] The media server is used to manage multiple media nodes and distribute and process media streams in specified media nodes; is used to interface with the video database to encrypt video storage; and is used to encrypt video media stream transmission.

[0014] In particular, the call information includes registered location information of the interrogatee, binding relationship information of the interrogator and the assigned interrogatee, and call information of both parties after the call, call duration, call time, and video storage address.

[0015] In particular, the 5G user terminal adopts an intelligent operating system mobile terminal based on Android or Harmony or IOS.

[0016] In particular, the interrogator server adopts an intelligent operating system based on Android or Harmony or IOS or Windows or Linux.

[0017] In particular, the specific method of the communication acceleration capability method for selecting routing information is:

[0018] (1) All media nodes that can be connected between the 5G user terminal and the interrogator server are divided into NODE node groups to form a node set S;

[0019] (2) Each NODE node in the node set S is detected to determine the NODE nodes that can be its neighbors;

[0020] (3) According to the packet loss and delay comprehensive parameters of the detection, the first ten NODE nodes with the lowest delay are reported to the media server as its neighbors;

[0021] (4) The media server determines the cost value between each NODE as the edge weight value of the set S by comprehensively considering the parameters of packet loss, delay, cost, and bandwidth, so that the node set S forms a directed graph G;

[0022] (5) The Dijkstra algorithm is used to calculate the directed graph G to calculate the shortest path P and the shortest path weight D of each NODE node; to ensure the audio and video quality, the optimal quality path tree is generated by the packet loss rate and delay and stored in the media server;

[0023] (6) When the 5G user terminal and the interrogator server are accessed, they respectively carry the nearest access optimal NODE information A and NODE information B, and the logical processing server determines the shortest path routing of the access points of the 5G user terminal and the interrogator server through the pre-calculated optimal quality path tree, then selects m routes according to the optimization principle, and finally the media server applies for a media forwarding port at each NODE on the m routing paths, and the media channel is opened.

[0024] An audio and video online communication method using the above audio and video online communication system, comprising the following steps:

[0025] The registration connection step of the interrogator server includes the interrogator login step of making the interrogated person log in the interrogator server and the heartbeat continuous maintenance step of making the interrogator server maintain an online state;

[0026] The alarm information establishment step, the 5G user terminal sends the interrogator information to the logical processing server;

[0027] The call plan generation step, the logical processing server queries the online interrogated personnel to the call plan database and allocates the interrogated personnel according to the idle condition;The logical processing server queries the best route to the media server;The media server returns the logical processing server by obtaining the best route information;The logical processing server returns the routing information to the 5G user terminal;

[0028] The call interrogated personnel and conversation step, the 5G user terminal calls the interrogated personnel through the routing information to the signaling server;The signaling server calls the called interrogated personnel service end according to the plan;The interrogated personnel service end is called after;The media server simultaneously establishes the media connection to the 5G user terminal and the interrogated personnel service end, and after the establishment is successful, the conversation is established and the media stream is continuously transmitted;

[0029] The information storage step, after the conversation ends, the media stream is disconnected, and the media server stores the video data to the video database;

[0030] The call sheet generation step, the signaling server establishes the call sheet information and sends it to the logical processing server;The logical processing server stores the call sheet information to the call plan database after receiving successfully, and ends after storing successfully.

[0031] In particular, the interrogated personnel sends a login request to the logical processing server through the interrogated personnel service end;The logical processing server confirms after authentication, and the location information is registered to the call plan database to generate a registration success instruction sent to the logical processing server;The logical processing server sends a login success instruction to the interrogated personnel service end after receiving the registration success instruction.

[0032] In particular, the heartbeat maintaining step is: the heartbeat maintaining, the interrogated personnel service end sends the REGISTER command to refresh the session state, and sends the REGISTER / expires=0 command to unregister the session.

[0033] The beneficial effects of the application are:

[0034] 1. The application combines 5G message with WebRTC, so that users can use the service by clicking after subscribing to 5G message, which is convenient and fast, and users do not need to download and install applications.

[0035] 2. Compared with the inefficient communication and long time-consuming inquiry of general telephone, the application adopts video communication, which is 70% higher in efficiency than telephone communication, and is more intuitive and time-saving.

[0036] 3. It provides convenience for deaf-mutes or people who cannot communicate by language, and can communicate through sign language or writing when language communication is not possible.

[0037] 4. Using communication acceleration technology, the delay and packet loss are reduced, and the smoothness of audio and video calls is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0039] Figure 1 The composition diagram of the audio and video online communication system of the embodiment of the present application;

[0040] Figure 2 The registration connection establishment process diagram of the interrogator server of the embodiment of the present application;

[0041] Figure 3 The connection process diagram of the 5G user terminal and the interrogator server of the embodiment of the present application;

[0042] Figure 4 The media node deployment situation diagram between the 5G user terminal and the interrogator server of the embodiment of the present application;

[0043] Figure 5 The directed graph G diagram and the optimal path route after calculation of the embodiment of the present application. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present application will be described in detail below with reference to the drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the protection scope of the present application can be more clearly and definitely defined.

[0045] It should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship commonly used when the product of the present application is used, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" and the like are only used for differentiation and cannot be understood as indicating or implying relative importance.

[0046] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0047] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0048] like Figure 1 As shown in this embodiment, an online audio and video communication system based on 5G messaging and WebRTC includes: a 5G user terminal, an interrogator server, a signaling server, a logic processing server, a media server, a call plan database, and a video storage database. The 5G user terminal is connected to the signaling server, media server, and logic processing server. The interrogator server is connected to the signaling server, media server, and logic processing server. The signaling server is connected to the logic processing server and media server. The logic processing server is connected to the call plan database. The media server is connected to the video database.

[0049] The 5G user terminal belongs to the person being questioned and is used to subscribe to 5G messages. It has a camera and microphone and internet access capabilities. The 5G user terminal is a mobile terminal based on a smart operating system such as Android, HarmonyOS, or iOS.

[0050] The interrogator server belongs to the online interrogated person and has camera and microphone functions as well as Internet access capability; it can maintain its online status by continuously sending registration requests to the signaling server or media server; the interrogator server adopts a smart operating system based on Android, HarmonyOS, iOS, Windows or Linux.

[0051] The logic processing server can assign an interrogated person to an interrogator for audio and video call, and generate a call plan containing interrogator information and interrogator server information; keep a heartbeat connection with the interrogator server, and update the location information in the call plan database when the connection is disconnected; interface with the 5G user terminal and the interrogator server, perform identity authentication and registration information; interface with the call plan database, generate a call relationship, and store call plan data when the call relationship is closed; interface with the media server, select the best media route through the communication acceleration capability method and establish routing information; interface with the signaling server, query the call plan information for the signaling server, and return to the call object; generate a bill information after the audio and video call ends, which includes the calling and called information and call information of the audio and video call and the media resource storage address.

[0052] The call plan database is used to store call information, so that the signaling server can query the call information through the call plan database, and the logic processing server can add, delete, modify and query the call plan through the call plan database; store the bill; the call information includes the registered location information of the interrogator, the binding relationship information of the interrogated person and the assigned interrogator, and the call information of both parties after the call ends, the call duration, the call time, and the video storage address.

[0053] The signaling server can interface with the logic processing server, establish a connection with the 5G user terminal and obtain the routing information of the interrogator server from the logic processing server and establish a connection therewith;

[0054] The video database is used to objectify and encrypt the audio and video call to store as audio and video storage data; the audio and video storage data correspond one-to-one with the call plan data; the logic processing server performs add, delete, modify and query operations on the audio and video database;

[0055] The media server is used to manage multiple media nodes and distribute and process media streams in specified media nodes; used to interface with the video database to encrypt video storage; used to encrypt video media stream transmission.

[0056] The specific method of the communication acceleration capability method for selecting routing information is:

[0057] (1) All media nodes that can be connected between the 5G user terminal and the interrogator server are divided into NODE node groups to form a node set S;

[0058] (2) Each NODE node in the node set S is detected to become a neighbor of the NODE node;

[0059] (3) According to the detected packet loss and delay comprehensive parameters, the ten lowest delay NODE nodes before the delay are reported as the neighbor to the media server;

[0060] (4) The media server determines the cost value between each NODE as the edge weight value of the set S by comprehensively considering the packet loss, delay, cost and bandwidth, so that the node set S forms a directed graph, as shown in Figure 4 .

[0061] (5) The Dijkstra algorithm is used to calculate the directed graph G, and the shortest path P and the shortest path weight D of each NODE node are calculated. To ensure the audio and video quality, the optimal quality path tree is generated by the packet loss rate and delay and stored in the media server.

[0062] (6) When the 5G user terminal (A) and the inquiry personnel service end (B) are accessed, the nearest optimal NODE information A (Sourcenode1, Sourcenode2, Sourcenode3) and B (Destnode1, Destnode2, Destnode3) are carried, and the logical processing server determines the shortest path routing of the access points of A and B by the pre-calculated optimal quality path tree, then selects m routes according to the optimization principle, and finally the media server applies for a media forwarding port at each NODE on the m routing paths, and the media channel is opened. As shown in Figure 5 , which is the optimal path from A to B.

[0063] As shown in Figure 2 , Figure 3 , an audio and video online communication method using the above audio and video online communication system, comprising the following steps:

[0064] The registration connection step of the inquiry personnel service end includes the inquiry personnel login step of making the person being interrogated to log in the inquiry personnel service end and the heartbeat continuous maintenance step of making the inquiry personnel service end maintain the online state;

[0065] The person being interrogated login step is: the person being interrogated sends a login request to the logical processing server through the inquiry personnel service end; after the logical processing server confirms the authentication, the location information is registered to the call plan database, and then a registration success instruction is generated and sent to the logical processing server; after the logical processing server receives the registration success instruction, a login success instruction is sent to the inquiry personnel service end.

[0066] The heartbeat continuous maintenance step is: the heartbeat is continuously maintained, the inquiry personnel service end sends a REGISTER command to refresh the session state, and a REGISTER / expires=0 command is sent to log out the session.

[0067] An inquiry information establishing step, in which the 5G user terminal sends inquiry information to the logical processing server;

[0068] A call plan generating step, in which the logical processing server queries the online inquired personnel from the call plan database and allocates the inquired personnel according to the idle condition; the logical processing server queries the optimal route from the media server; the media server returns the logical processing server by obtaining the optimal route information; and the logical processing server returns the routing information to the 5G user terminal;

[0069] A call inquired personnel and conversation step, in which the 5G user terminal calls the inquired personnel through the routing information from the signaling server; the signaling server calls the inquired personnel service end according to the plan; the inquired personnel service end is called; the media server simultaneously establishes the media connection between the 5G user terminal and the inquired personnel service end, and after the establishment is successful, the conversation is established and the media stream is continuously transmitted;

[0070] An information storing step, in which after the conversation is ended, the media stream is disconnected, and the media server stores the video data to the video database;

[0071] A bill generating step, in which the signaling server establishes the bill information and sends it to the logical processing server; the logical processing server stores the bill information to the call plan database after the reception is successful, and the process is ended after the storage is successful.

[0072] Although the embodiments of the present application are described in conjunction with the drawings, the patent owner can make various modifications or changes within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the present application, and should be within the protection scope of the present application. The principles and embodiments of the present application are described in this paper, and the above examples are only used to help understand the method of the present application and its core idea. The above is only the preferred embodiment of the present application, it should be pointed out that due to the limited nature of the expression, there are infinite specific structures, for ordinary skilled in the art, without departing from the principles of the present application, can make a number of improvements, decoration or change, or the above technical features can be combined in an appropriate way; these improvements, decoration, change or combination, or without improvement, the concept and technical scheme of the present application is directly applied to other occasions, should be regarded as the protection scope of the present application.

Claims

1. An audio and video online communication system based on the combination of 5G message and WebRTC, comprising: a 5G user terminal, an interrogator server, a signaling server, a logic processing server, a media server, a call plan database, and a video storage database; the 5G user terminal is connected to the signaling server, the media server, and the logic processing server respectively; the interrogator server is connected to the signaling server, the media server, and the logic processing server respectively; the signaling server is connected to the logic processing server and the media server respectively; the logic processing server is connected to the call plan database; the media server is connected to the video database; the 5G user terminal belongs to an interrogator and is used for subscribing to 5G messages and has camera and microphone functions and Internet access capability; the interrogator server belongs to an online interrogatee and has camera and microphone functions and Internet access capability; it can maintain an online state by continuously initiating a registration request to the signaling server or the media server; the logic processing server can assign interrogated persons for audio and video interrogation and generate a call plan, which contains interrogator information and interrogator server information; it is used for maintaining a heartbeat connection with the interrogator server and updating location information in the call plan database when the connection is disconnected; it can interface the 5G user terminal and the interrogator server, perform identity authentication and registration information, interface the call plan database, generate a call relationship, and store call plan data when the call relationship is closed; it can interface the media server, select the best media route through a communication acceleration capability method, and establish routing information; it can interface the signaling server, query call plan information for the signaling server, and return to the call object; it can generate a call record information after the audio and video call ends, which includes the calling and called information and call information of the audio and video call and a media resource storage address; the call plan database is used for storing call information, enabling the signaling server to query call information through the call plan database, and enabling the logic processing server to add, delete, modify, and query call plans through the call plan database; store call records; the signaling server can interface the logic processing server, establish a connection with the 5G user terminal when calling, and obtain routing information of the interrogator server from the logic processing server and establish a connection therewith; the video database is used for object-oriented encryption storage of audio and video calls as audio and video storage data; the audio and video storage data correspond one-to-one with call plan data; the logic processing server performs add, delete, modify, and query operations on the audio and video database; the media server is used for managing multiple media nodes and distributing and processing media streams in specified media nodes; it is used for interfacing the video database to encrypt video storage; it is used for encrypting video media stream transmission.

2. The audio and video online communication system based on the combination of 5G message and WebRTC according to claim 1, characterized in that: The call information includes the registered location information of the interrogated person, the binding relationship information of the interrogator and the assigned interrogated person, and the call information of both parties after the call ends, the call duration, the call time, and the video storage address.

3. The audio and video online communication system based on the combination of 5G message and WebRTC according to claim 1, characterized in that: The 5G user terminal adopts an intelligent operating system mobile terminal based on Android or Harmony or IOS.

4. The audio and video online communication system based on the combination of 5G message and WebRTC according to claim 1, characterized in that: The inquiring personnel server adopts an intelligent operating system based on Android or Harmony or IOS or windows or linux.

5. The audio and video online communication system based on the combination of 5G message and WebRTC according to claim 1, characterized in that: The specific method for selecting route information by the communication acceleration capability method is: (1) dividing all media nodes capable of connecting between the 5G user terminal and the inquiring personnel server as NODE nodes to form a node set S; (2) detecting each NODE node in the node set S to determine the NODE nodes that can possibly become neighbors; (3) according to the packet loss and delay comprehensive parameters of the detection, reporting the first ten NODE nodes with the lowest delay as the neighbors of the media server; (4) determining the cost value between each NODE as the edge weight value of the set S by the media server based on the parameters of packet loss, delay, cost and bandwidth, so that the node set S forms a directed graph G; (5) calculating the shortest path P and the shortest path weight D of each NODE node by using the Dijkstra algorithm on the directed graph G; to ensure the audio and video quality, the optimal quality path tree is generated based on the packet loss rate and delay and stored in the media server; (6) when the 5G user terminal and the inquiring personnel server are accessed, the NODE information A and the NODE information B of the nearest access optimal are carried respectively, the shortest path route of the access points of the 5G user terminal and the inquiring personnel server is determined by the logical processing server through the pre-calculated optimal quality path tree, then m routes are selected according to the optimization principle, and finally the media forwarding port of each NODE on the m route paths is applied by the media server, and the media channel is opened.

6. An audio and video online communication method using the audio and video online communication system of any one of claims 1-5, comprising the following steps: a registration and connection step of the inquiring personnel server, including a being-inquired personnel login step of making the being-inquired personnel log in the inquiring personnel server and a heartbeat continuous keeping step of making the inquiring personnel server maintain an online state; an inquiring information establishing step, in which the 5G user terminal sends inquiring information to the logical processing server; a call plan generating step, in which the logical processing server queries the online being-inquired personnel from the call plan database and allocates the being-inquired personnel according to the idle condition; the logical processing server queries the best route from the media server; the media server returns the best route information to the logical processing server; the logical processing server returns the route information to the 5G user terminal; a calling being-inquired personnel and communication step, in which the 5G user terminal calls the inquiring personnel through the route information; the signaling server calls the inquiring personnel according to the plan; the inquiring personnel server answers the call; the media server simultaneously establishes media connection between the 5G user terminal and the inquiring personnel server, and after the establishment is successful, the communication is established and the media stream is continuously transmitted; an information storage step, in which after the communication is ended, the media stream is disconnected, and the media server stores the video data in the video database; a call record generating step, in which the signaling server establishes the call record information and sends it to the logical processing server. The logic processing server receives the successful storage of the call record information into the call plan database, and ends after the storage.

7. The audio and video online communication method of the audio and video online communication system according to claim 6, characterized in that: The interrogated person login step is that the interrogated person sends a login request to the logic processing server through the interrogator server. After the logic processing server confirms the authentication, the location information is registered into the call plan database to generate a registration success instruction and send it to the logic processing server. The logic processing server receives the registration success instruction and sends a login success instruction to the interrogator server.

8. The audio and video online communication method of the audio and video online communication system according to claim 6, characterized in that: The heartbeat continuous maintenance step is that the heartbeat is continuously maintained, the interrogator server sends a REGISTER command to refresh the session state, and sends a REGISTER / expires=0 command to log out the session.

Images