A WebRTC audio and video call function integration method, terminal device and storage medium

Abstract

This invention relates to a method, terminal device, and storage medium for integrating WebRTC audio and video call functions. The method includes: building an SDK based on WebRTC; the SDK includes a core module, SDK-core; the SDK-core internally includes a call instance, Phone; and the Phone instance includes call-related methods and an On method for registering and monitoring events. The method uses WebSocket technology to send and receive SIP packets to establish a connection with an IP-PBX server, enabling SIP instance registration and the sending and receiving of SIP messages between the client and the IP-PBX server. The collected call data is then converted into a visual interface, providing convenient user interaction.

Description

Technical Field

[0001] The present invention relates to the field of audio and video calls, and in particular to a WebRTC audio and video call function integration method, terminal equipment, and storage medium. Background Art

[0002] Currently, web applications in the industry use WebRTC technology to implement common call scenarios such as audio and video calls, call transfers, call recording, and DTMF (Dual Tone Multi Frequency). However, the following problems may arise in actual web application development:

[0003] (1) Limited intercommunication with various types of terminals: Applications implemented using only WebRTC can only communicate with other applications implemented using WebRTC, and cannot communicate with devices such as IP phones and door phones.

[0004] (2) High development and learning costs: Front-end development of call functions based on the combination of SIP protocol and WebRTC requires familiarity with the audio and video communication principles of WebRTC technology. It also requires a deep understanding of the relevant standards and specifications of the SIP protocol, the implementation principles and workflow of the SIP protocol stack, and protocols associated with SIP, such as SDP, RTP, RTCP, and video codecs like VP8 and H264.

[0005] (3) Poor versatility: It cannot be ported and used in other web applications, and cannot be reused or promoted at will.

[0006] (4) Long development cycle: Developers need to complete SIP registration, handle various call processes, implement call UI, and other time-consuming development tasks. Summary of the Invention

[0007] In order to solve the above problems, the present invention proposes a WebRTC audio and video call function integration method, terminal device and storage medium.

[0008] The specific plan is as follows:

[0009] A method for integrating WebRTC audio and video call functions, comprising:

[0010] The SDK is built based on WebRTC. The SDK includes the core module sdk-core. The sdk-core contains the call instance phone. The call instance phone includes call-related methods and the on method for registering and listening for events, implemented based on Node.js's events.EventEmitter. When creating the call method within the call-related methods, sdk-core creates a Session object and stores call-related data and all other call-related methods within the Session.

[0011] (1) SDK registration process:

[0012] S101: After the third-party application imports the SDK, it sends call-related parameters to the SDK-core;

[0013] S102: SDK-core registers with the IP-PBX based on the received call-related parameters. After successful registration, it returns the call instance phone and monitors call events based on the on method provided by the call instance phone.

[0014] (2) Call process:

[0015] S201: When the call method provided by the call instance phone is called, sdk-core calls the call method in JsSIP based on the call method;

[0016] S202: JsSIP creates an RTCPeerConnection instance of WebRTC through the call method and creates an offer through the createOffer method of the RTCPeerConnection instance;

[0017] S203: After JsSIP obtains the SDP in the offer, it encapsulates the SDP into the body of the INVITE packet in the SIP protocol format and sends it to the IP-PBX via websocket;

[0018] S204: When JsSIP receives the SIP packet with a 200 OK response from the IP-PBX, the call is successfully established. After parsing the SDP carried in the SIP packet, it sets the SDP as the answer to the RTCPeerConnection instance. WebRTC then performs an ICE connectivity check based on the information carried in the SDP.

[0019] S205: After the ICE connectivity check is completed, the addresses and ports of the sending and receiving streams of the two parties in the call are determined. Based on the addresses and ports of the sending and receiving streams of the two parties in the call, WebRTC and the IP-PBX send streams to each other;

[0020] S206: When the hang-up method in the Session is called, sdk-core calls the terminate method of JsSIP based on the hang-up method and sends a BYE packet to the IP-PBX;

[0021] S207: The IP-PBX receives the BYE packet and responds with 200OK.

[0022] S208: After JsSIP receives the SIP packet with a 200OK response from the IP-PBX, it triggers the end call event, removes the stream, and disconnects the RTCPeerConnection connection, ending the call.

[0023] Furthermore, in step S102, the method for sdk-core to register with the IP-PBX is as follows: sdk-core creates a websocket and connects. After the connection is successful, the websocket instance is set as a parameter to the constructor for initializing JsSIP and a JsSIP instance is created. A registration request is initiated to the SIP server by calling the start method of the JsSIP instance.

[0024] Furthermore, the sdk also includes an interface module sdk-ui. In step S101, the third-party application needs to first send call-related parameters to sdk-ui; sdk-ui sends the call-related parameters to sdk-core.

[0025] Furthermore, the SDK also includes an interface module SDK-UI. The third-party application sends interface-related parameters to the SDK-UI. After successful registration in step S102, the SDK-UI creates a visual interface based on the interface-related parameters.

[0026] Furthermore, the visualization page includes the following three modules:

[0027] (1) Incoming call window module: used to convert the data in the Session into an interface display when an incoming call is received;

[0028] (2) Call interface module: used for displaying during a call, including the calling user's identity information, audio information, and / or video information;

[0029] (3) Dial pad module: used to input and dial numbers, and make calls by calling the call method in the call instance.

[0030] Furthermore, the call-related methods also include any number of methods for answering, methods for hanging up, methods for blind transfer, methods for consultative transfer, methods for holding, methods for releasing hold, methods for sending DTMF, methods for muting, methods for unmuting, methods for starting recording, methods for pausing recording, methods for converting audio calls to video calls, methods for converting video calls to audio calls, and methods for renegotiating calls.

[0031] Furthermore, it also includes: when different call functions need to be implemented, the corresponding method in the call-related method in the call instance phone is called, and sdk-core calls the corresponding method in JsSIP based on the called method to implement the required call function.

[0032] A terminal device with integrated WebRTC audio and video call functions includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above are implemented in an embodiment of the present invention.

[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described above in an embodiment of the present invention.

[0034] The present invention adopts the above technical solution, establishes a connection with the IP-PBX server by sending and receiving SIP packets through WebSocket technology, realizes SIP instance registration and SIP message sending and receiving between the client and the IP-PBX server, and converts the collected call data into a visual interface to provide convenient interaction for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure shows the internal relationship diagram of sdk-core in the first embodiment of the present invention.

[0036] Figure 2 The figure shows a communication diagram of the integrated SDK in this embodiment.

[0037] Figure 3 Shown is a flow chart of the conversation phase in this embodiment. DETAILED DESCRIPTION

[0038] To further illustrate various embodiments, the present invention provides accompanying drawings. These drawings form part of the present disclosure and are primarily used to illustrate the embodiments and, in conjunction with the relevant description in the specification, to explain the operating principles of the embodiments. By referring to these drawings, those skilled in the art will be able to understand other possible implementations and the advantages of the present invention.

[0039] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.

[0040] Example 1:

[0041] An embodiment of the present invention provides a method for integrating WebRTC audio and video call functions, the method comprising:

[0042] Build SDK based on WebRTC. SDK is divided into two parts: kernel module sdk-core and interface module sdk-ui.

[0043] sdk-core primarily handles call-related logic, including receiving incoming and outgoing calls, modifying call status, and exposing call-related methods to external programs such as sdk-ui and third-party applications. sdk-core is an independent module. Third-party applications that do not want to use sdk-ui can use sdk-core independently and implement their own interface interactions.

[0044] sdk-ui is used to visualize call status events and provide a convenient and user-friendly interactive interface. sdk-ui must be used in conjunction with sdk-core.

[0045] The SDK supports installation via npm (Node.js package manager) and import via umd. Users can choose the appropriate method for their specific scenario. Users can directly use the integrated SDK (including sdk-core and sdk-ui) or use a solution that only includes sdk-core. The main differences are:

[0046] (1) When users have less demand for special customized call scenarios, the SDK integrates SDK-core and provides a rich visual interface. Users only need to configure the initialization parameters, and the SDK will handle other issues.

[0047] (2) If users have many special requirements for call scenarios that cannot be met by the existing interface interaction of sdk-ui, they can use sdk-core to develop.

[0048] SDK-core includes a phone instance. This instance includes call-related methods and an on method, implemented based on Node.js's events.EventEmitter, for registering and listening for events. SDK-core uses the on method to register and listen for events, including incoming calls, registration status changes, call start, and call end. These events are used to modify functional view components within the visual interface.

[0049] In this embodiment, the call-related methods include the outgoing call method, the answer method, the hangup method, the blind transfer method, the attended transfer method, the hold method, the unhold method, the DTMF sending method, the mute method, the unmute method, the start recording method, the pause recording method, the audioToVideo method for converting an audio call to a video call, the videoToAudio method for converting a video call to an audio call, and the renegotiate method for renegotiating a call. Those skilled in the art can adjust the call-related methods included in the call instance "phone" according to actual usage requirements. The above call-related methods are all encapsulated by corresponding methods in the JsSIP tool library and are used to call corresponding methods in the JsSIP tool library. For example, the outgoing call method "call" is used to call the call method in the JsSIP tool library.

[0050] When sdk-core creates a call method in the call-related method, it also creates a Session object and stores the call-related data and other methods other than the call method in the session. For example, Figure 1 shown.

[0051] (1) SDK registration process:

[0052] S101: After the third-party application imports the SDK, it sends call-related parameters to the SDK-core.

[0053] The third-party application can be a browser or desktop application that supports WebRTC.

[0054] In this embodiment, the call-related parameters include at least a user name, a signature key, and an IP-PBX address.

[0055] When the third-party application imports an integrated SDK, the communication method is as follows Figure 2 As shown, a third-party application client imports the SDK-UI, which establishes communication with the IP-PBX through the SDK-core. When passing parameters, the third-party application first sends interface and call parameters to the SDK-UI. The SDK-UI generates the corresponding visualization interface based on the interface parameters and sends the call parameters to the SDK-core. If the third-party application imports a separate SDK-core, the third-party application directly sends the call parameters to the SDK-core, and the third-party application builds the required visualization interface.

[0056] S102: sdk-core registers with the IP-PBX based on the received call-related parameters. After successful registration, it returns the call instance phone and monitors call events based on the on method provided by the call instance phone.

[0057] In this embodiment, the method for sdk-core to register with the IP-PBX is as follows: sdk-core creates a websocket and connects. After the connection is successful, the websocket instance is set as a parameter to the constructor that initializes JsSIP and creates a JsSIP instance. A registration request is initiated to the SIP server by calling the start method of the JsSIP instance.

[0058] When the third-party application imports an integrated SDK, the call instance phone is returned to the sdk-ui; when the third-party application imports a separate sdk-core, the call instance phone is returned to the third-party application.

[0059] When a third-party application imports an integrated SDK, SDK-UI will not start creating the visual interface until the application is successfully registered. The successful registration is determined by SDK-UI receiving the returned call instance phone.

[0060] The visualization interface created by sdk-ui in this embodiment includes the following three modules:

[0061] (1) Incoming call window module: used to convert the data in the Session into an interface display when receiving an incoming call, and provides audio and video button methods for answering the call and hanging up the call.

[0062] (2) Call interface module: used for display during a call, the display content includes the call user identity information, audio information, and / or video information, and may also include common call operation function buttons.

[0063] (3) Dial pad module: used to input and dial numbers, and make calls by calling the call method in the call instance.

[0064] (2) Call process (such as Figure 3 shown):

[0065] S201: When the call method provided by the call instance phone is called, sdk-core calls the call method in JsSIP based on the call method.

[0066] Before sdk-core calls the call method in JsSIP, it also includes: sdk-core checks the number being called and the status of the audio and video devices of the terminal corresponding to the third-party application, and calls the call method in JsSIP when the check passes.

[0067] S202: JsSIP creates an RTCPeerConnection instance of WebRTC through the call method, and creates an offer based on the outgoing call type through the createOffer method of the RTCPeerConnection instance.

[0068] When creating an offer, create an offer of the corresponding type based on the outgoing call type. The outgoing call type includes audio type (voice call) and video type (video call).

[0069] S203: After JsSIP obtains the SDP (session descriptor, which describes the call media information, such as whether it is an audio or video call and which audio and video codec to use) in the offer, it encapsulates the SDP into the body of the INVITE packet in the SIP protocol format and sends it to the IP-PBX via websocket (a protocol for full-duplex communication over a single TCP connection).

[0070] S204: When JsSIP receives the SIP packet with a 200 OK response from the IP-PBX, the call is successfully established. After parsing the SDP contained in the SIP packet, JsSIP sets the SDP as the answer to the RTCPeerConnection instance. WebRTC then performs an ICE connectivity check based on the information contained in the SDP.

[0071] After receiving the INVITE, the IP-PBX will recover in the order of Trying, Ringing, and 200OK. When it replies with 200OK, it indicates that the other party has answered the call and the call is successfully established.

[0072] S205: After the ICE connectivity check is completed, the addresses and ports of the sending and receiving streams of the two parties are determined. Based on the addresses and ports of the sending and receiving streams of the two parties, WebRTC and the IP-PBX send streams to each other, and users at both ends can hear audio or see video.

[0073] After the addresses and ports of the sending and receiving streams of both parties in the call are determined, the call timing can be started in the call window on the visual interface.

[0074] S206: When the hang-up method in the Session is called, the sdk-core calls the terminate method of JsSIP based on the hang-up method and sends a BYE packet to the IP-PBX.

[0075] S207: After receiving the BYE packet, the IP-PBX responds with a 200 OK.

[0076] S208: After JsSIP receives the SIP packet with a 200OK response from the IP-PBX, it triggers the end call event, removes the stream, and disconnects the RTCPeerConnection connection, ending the call.

[0077] When a user needs to perform a call function, they can do so by selecting the corresponding function button on the visual interface provided by a third-party application or the SDK-UI. When a function button is selected, the third-party application or SDK-UI calls the corresponding call-related method in the call instance phone. Based on the called method, the SDK-core calls the corresponding method in JsSIP to implement the desired call function. For example, the call method and hang-up method mentioned in the call flow above can also be called upon receiving an incoming call event, or after a call is established, any call-related method can be called for blind transfer, consultative transfer, hold, unhold, mute, unmute, start recording, pause recording, and so on.

[0078] The embodiments of the present invention have the following beneficial effects:

[0079] (1) Ability to communicate with various types of terminals: Because it is developed based on the SIP protocol, it can communicate with communication devices that support the SIP protocol.

[0080] (2) Good versatility: The SDK can be imported and used in current web applications.

[0081] (3) Short development cycle: The SDK is ready to use out of the box, enabling visualization of audio and video calls and common call scenarios. Third-party users do not need to have an in-depth understanding of WebRTC technology, SIP, and related protocols, nor do they need to implement a call visualization interface, saving development time.

[0082] (4) Low development threshold: The SDK has highly encapsulated the call function. Users do not need to pay attention to the initialization process such as SIP registration. When initializing the application, they only need to pass in the necessary three parameters to complete the initialization. Using visual initialization, users can use the call without any further operations after the initialization is completed, which greatly reduces the development burden.

[0083] (5) Support for configurable UI styles: Third-party users can modify the display position of the visual interface and the UI style of some pages according to the usage scenario.

[0084] (6) Timeliness and security: The SDK uses an encrypted TLS connection to establish a WebSocket, which ensures the timeliness of call messages while also ensuring the security of the messages.

[0085] (7) Wide range of applications: The SDK is a communication solution that combines SIP and WebRTC. Through IP-PBX forwarding, it supports not only Internet phone communications but also communication with traditional IP phones. In addition, the SDK has strong cross-platform capabilities. It can be used on mobile phones (Apple, Android) and computers (Windows, Mac, Ubuntu) by simply providing an environment that can run JS and the WebRTC module.

[0086] Example 2:

[0087] The present invention also provides a terminal device integrated with WebRTC audio and video call functions, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiment of the first embodiment of the present invention are implemented.

[0088] Furthermore, as an executable solution, the WebRTC audio and video call function integrated terminal device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The WebRTC audio and video call function integrated terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the composition structure of the above-mentioned WebRTC audio and video call function integrated terminal device is only an example of a WebRTC audio and video call function integrated terminal device, and does not constitute a limitation on the WebRTC audio and video call function integrated terminal device. It may include more or fewer components than the above-mentioned components, or a combination of certain components, or different components. For example, the WebRTC audio and video call function integrated terminal device may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0089] Furthermore, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices. The general-purpose processor may be a microprocessor or any conventional processor. The processor serves as the control center of the WebRTC audio and video call function integrated terminal device, connecting various components of the entire WebRTC audio and video call function integrated terminal device using various interfaces and lines.

[0090] The memory can be used to store the computer programs and / or modules, and the processor implements various functions of the WebRTC audio and video call function integrated terminal device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0091] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0092] If the module / unit integrated into the terminal device with the WebRTC audio and video call function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM) and software distribution medium, etc.

[0093] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A method for integrating WebRTC audio and video call functions, characterized in that: include: Build SDK based on WebRTC; The SDK includes the kernel module sdk-core; the SDK-core includes the call instance phone; The call instance phone includes call-related methods and the on method for registering and listening events based on events.EventEmitter in nodejs; When creating a call method in a call-related method, sdk-core creates a Session object and stores call-related data and other methods other than the call method in the Session; (1) SDK registration process: S101: After the third-party application imports the SDK, it sends call-related parameters to the SDK-core; S102: SDK-core registers with the IP-PBX based on the received call-related parameters. After successful registration, it returns the call instance phone and monitors call events based on the on method provided by the call instance phone. (2) Call process: S201: When the call method provided by the call instance phone is called, sdk-core calls the call method in JsSIP based on the call method; S202: JsSIP creates an RTCPeerConnection instance of WebRTC through the call method and creates an offer through the createOffer method of the RTCPeerConnection instance; S203: After JsSIP obtains the SDP in the offer, it encapsulates the SDP into the body of the INVITE packet in the SIP protocol format and sends it to the IP-PBX via websocket; S204: When JsSIP receives the SIP packet with a 200 OK response from the IP-PBX, the call is successfully established. After parsing the SDP carried in the SIP packet, it sets the SDP as the answer to the RTCPeerConnection instance. WebRTC then performs an ICE connectivity check based on the information carried in the SDP. S205: After the ICE connectivity check is completed, the addresses and ports of the sending and receiving streams of the two parties in the call are determined. Based on the addresses and ports of the sending and receiving streams of the two parties in the call, WebRTC and the IP-PBX send streams to each other; S206: When the hang-up method in the Session is called, sdk-core calls the terminate method of JsSIP based on the hang-up method and sends a BYE packet to the IP-PBX; S207: The IP-PBX receives the BYE packet and responds with 200OK. S208: After JsSIP receives the SIP packet with a 200OK response from the IP-PBX, it triggers the end call event, removes the stream, and disconnects the RTCPeerConnection connection, ending the call.

2. The WebRTC audio and video call function integration method according to claim 1, characterized in that: In step S102, the method for sdk-core to register with the IP-PBX is as follows: sdk-core creates a websocket and connects. After the connection is successful, the websocket instance is set as a parameter to the constructor that initializes JsSIP and creates a JsSIP instance. A registration request is initiated to the SIP server by calling the start method of the JsSIP instance.

3. The WebRTC audio and video call function integration method according to claim 1, characterized in that: The sdk also includes an interface module sdk-ui. In step S101, the third-party application needs to first send call-related parameters to the sdk-ui; the sdk-ui sends the call-related parameters to the sdk-core.

4. The WebRTC audio and video call function integration method according to claim 1, characterized in that: The sdk also includes an interface module sdk-ui. The third-party application sends interface-related parameters to the sdk-ui. After successful registration in step S102, the sdk-ui creates a visual interface based on the interface-related parameters.

5. The WebRTC audio and video call function integration method according to claim 4, characterized in that: The visualization page includes the following three modules: (1) Incoming call window module: used to convert the data in the Session into an interface display when an incoming call is received; (2) Call interface module: used for displaying during a call, including the calling user's identity information, audio information, and / or video information; (3) Dial pad module: used to input and dial numbers, and make calls by calling the call method in the call instance.

6. The WebRTC audio and video call function integration method according to claim 1, characterized in that: Call-related methods also include any number of methods for answering, methods for hanging up, methods for blind transfer, methods for consultative transfer, methods for holding, methods for releasing hold, methods for sending DTMF, methods for muting, methods for unmuting, methods for starting recording, methods for pausing recording, methods for converting an audio call to a video call, methods for converting a video call to an audio call, and methods for renegotiating a call.

7. The WebRTC audio and video call function integration method according to claim 1, characterized in that: Also includes: When different call functions need to be implemented, the corresponding methods in the call-related methods in the call instance phone are called. SDK-core calls the corresponding methods in JsSIP based on the called methods to implement the required call functions.

8. A terminal device integrating WebRTC audio and video call functions, characterized by: The method comprises a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images