NLP real-time testing method, device, medium and equipment based on freeswitch and webrtc technology

Abstract

The application relates to an NLP real-time testing method, device, medium and equipment based on freeswitch and WebRTC technology, wherein the method comprises the following steps: receiving a preset account transmitted by WebRTC in a browser through freeswitch, and obtaining a valid user account corresponding to the preset account; receiving a start instruction sent by the browser, establishing a two-way communication channel between the freeswitch and the browser based on the valid user account; obtaining text information corresponding to a voice stream transmitted by the WebRTC channel through the freeswitch based on the two-way communication channel, analyzing the text information through NLP, and obtaining a context conversation corresponding to the text information; and sending the context conversation to the browser through the freeswitch, so that the browser displays the context conversation in a text conversation box. The application has the effect of improving the efficiency of NLP testing.

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to a method, apparatus, medium, and device for real-time NLP testing based on FreeSwitch and WebRTC technologies. Background Technology

[0002] The Public Switched Telephone Network (PSTN), a commonly used older telephone system, is a global circuit-switched voice communication network. However, the PSTN incurs additional communication costs and is significantly affected by signal quality and policy.

[0003] When a traditional phone call is connected, the context of the conversation between the two parties cannot be seen in real time. In order to see the context of the conversation in real time, it is usually necessary to implement a normal Natural Language Processing (NLP) context dialogue. Before implementing a normal NLP context dialogue, a large amount of training and testing of NLP is required through PSTN, which results in low efficiency of NLP testing. Summary of the Invention

[0004] To improve the efficiency of NLP testing, this application provides a method, apparatus, medium, and device for real-time NLP testing based on FreeSwitch and WebRTC technologies.

[0005] The first aspect of this application provides a method for real-time NLP testing based on FreeSwitch and WebRTC technologies, specifically including:

[0006] The system receives a preset account transmitted via WebRTC in the browser through FreeSWITCH and obtains the valid user account corresponding to the preset account.

[0007] Receive the start command sent by the browser and establish a two-way communication channel between the FreeSWITCH and the browser based on the valid user account;

[0008] Based on the bidirectional communication channel, the text information corresponding to the voice stream transmitted through the bidirectional communication channel is obtained through the freeswitch, and the text information is parsed through NLP to obtain the context dialogue corresponding to the text information;

[0009] The context dialogue is sent to the browser via the FreeSwitch, so that the browser displays the context dialogue in a text dialog box.

[0010] By employing the above technical solution, WebRTC, integrated in the browser, sends a preset account to FreeSwitch on the server. FreeSwitch then registers and authenticates the preset account, making it a valid user account. Next, FreeSwitch receives a start command from the browser, bridges the valid user account existing in the context, and establishes communication between FreeSwitch and the valid user account. This establishes a bidirectional communication channel between FreeSwitch and the browser. The browser and FreeSwitch acquire the audio stream transmitted through this bidirectional communication channel and convert it into corresponding text information. Finally, FreeSwitch bridges the communication channel with NLP, sending the text information to NLP for parsing to obtain the context dialogue corresponding to the text information. This text is then sent to the browser through the bidirectional communication channel and displayed as a dialog text box on the browser page. This bidirectional communication channel enables real-time testing of NLP through the transmission of media streams (audio and text), eliminating the need for extensive PSTN testing and significantly improving NLP testing efficiency.

[0011] Optionally, the step of receiving a preset account transmitted via WebRTC in a browser through FreeSWITCH and obtaining a valid user account corresponding to the preset account includes:

[0012] Based on the SIP protocol, a preset account is received from WebRTC transmission in the browser via FreeSWITCH;

[0013] Register the preset account through FreeSWITCH to obtain the registered account;

[0014] The registered account is authenticated using FreeSwitch via a 401 redirect. If the 401 authentication is successful, the registered account is confirmed as a valid user account.

[0015] By adopting the above technical solution, the FreeSWITCH in the server receives the preset account sent by WebRTC based on the SIP protocol. Then, FreeSWITCH registers the preset account to obtain a registered account. Finally, when the user logs in to the browser with the registered account, FreeSWITCH performs 401 authentication on the registered account to verify the user identity. If the 401 authentication is successful, it means that the identity verification is passed, and the registration of a valid user is completed on FreeSWITCH, that is, a valid user account is obtained, which facilitates the establishment of bidirectional communication between FreeSWITCH and the browser.

[0016] Optionally, receiving the start command sent by the browser and establishing a bidirectional communication channel between FreeSWITCH and the browser based on the valid user account includes:

[0017] The system receives the start command sent by the browser via WebSocket, and the FreeSWITCH initiates a media call request to the browser.

[0018] After initiating a media call request to the browser through the FreeSwitch, the system receives the call instruction sent by the browser and bridges the valid user account. A bidirectional communication channel between the FreeSwitch and the browser is then established through SIP signaling negotiation.

[0019] By adopting the above technical solution, once a valid user account is identified, FreeSwitch receives the NLP test start command from the browser. In response to this command, FreeSwitch initiates a media call request to the browser, indicating that the browser and FreeSwitch can conduct a formal voice call. Then, after receiving the call command from the browser, the voice call officially begins, bridging the valid user account and utilizing FreeSwitch's B2B functionality to establish communication with the valid user account. This quickly establishes a bidirectional communication channel between FreeSwitch and the browser, facilitating subsequent media stream transmission through this channel.

[0020] Optionally, the bidirectional communication channel includes a WebRTC channel. Based on the bidirectional communication channel, the text information corresponding to the voice stream transmitted through the freeswitch is obtained, and the text information is parsed using NLP to obtain the contextual dialogue corresponding to the text information, including:

[0021] The RTP voice stream transmitted through the WebRTC channel is obtained through the freeswitch;

[0022] The RTP voice stream is converted into corresponding text information using ASR technology;

[0023] The text information is encapsulated to obtain encapsulated text information;

[0024] The encapsulated text information is sent to NLP for parsing via FreeSWITCH to obtain the context dialogue corresponding to the text information.

[0025] By adopting the above technical solution, after establishing a bidirectional communication channel between FreeSWITCH and the browser, the browser sends the RTP voice stream to FreeSWITCH through the WebRTC channel in the bidirectional communication channel. FreeSWITCH then uses ASR (Automatic Speech-to-Text) technology to convert the RTP voice stream into text information. Next, FreeSWITCH encapsulates the text information to ensure security during transmission. Finally, the encapsulated text information is sent to NLP (Network Logic Programming), where NLP performs context parsing to obtain the context dialogue corresponding to this text information, thus enabling real-time acquisition of the context dialogue corresponding to the voice stream sent by the browser.

[0026] Optionally, the bidirectional communication channel further includes a WebSocket channel, and after converting the RTP voice stream into corresponding text information using ASR technology, it further includes:

[0027] The FreeSwitch program sends the RTP audio stream to the browser via the WebRTC channel, allowing the browser to play the RTP audio stream; or

[0028] The FreeSWITCH sends the text information to the browser via the WebSocket channel, so that the browser can play the text information.

[0029] By adopting the above technical solution, since the WebRTC channel supports the transmission of voice streams and the WebSocket channel supports the transmission of text and binary data, FreeSwitch converts the RTP voice stream into text information and sends the RTP voice stream to the browser for playback via the WebRTC channel in the bidirectional communication channel. This allows users to hear real-time conversational voice messages in the browser, or send text information to the browser via the WebSocket channel in the bidirectional communication channel, allowing users to read the corresponding text in the browser in real time.

[0030] Optionally, the bidirectional communication channel further includes a WebSocket channel, and the step of sending the context dialog to the browser via the FreeSWITCH so that the browser displays the context dialog in a text dialog box includes:

[0031] The context dialogue is obtained by the FreeSwitch from the NLP through the NLP interface;

[0032] The context dialogue is encapsulated to obtain an Event;

[0033] Based on the MQ message mechanism, the Event is sent to the browser through the WebSocket channel, so that the browser displays the context dialog in a text dialog box.

[0034] By adopting the above technical solution, NLP parses text information into contextual dialogue, and then uses FreeSwitch to call the NLP interface to conveniently and securely obtain the contextual dialogue from NLP. Next, FreeSwitch encapsulates the contextual dialogue into Events, and based on the Message Queuing (MQ) mechanism, converts the Events into MQ messages and sends them in a queue, effectively improving system performance and reducing response time. Finally, the MQ messages converted from the Events are sent to the browser through a bidirectional communication channel (WebSocket), allowing the dialogue text forming the contextual logic (contextual dialogue) to be displayed in a dialogue text box on the browser page, thus enabling real-time and stable testing of NLP and improving NLP testing efficiency.

[0035] Optionally, the step of sending the Event to the browser via the WebSocket channel based on the MQ message mechanism, so that the browser displays the context dialog in a text dialog box, includes:

[0036] Based on the MQ message mechanism, the set of WebSocket channels recorded by the server is obtained through the FreeSWITCH, and the target WebSocket channel for the start command transmission is selected from the set of WebSocket channels.

[0037] The Event is sent to the browser via the target WebSocket channel, so that the browser displays the context dialog in a text dialog box.

[0038] By employing the above technical solution, the Java application on the server obtains a set of WebSocket channels, as the Java application automatically records all WebSocket channels established through the browser. Next, it checks if the WebSocket channel used to initiate the command transmission exists in the WebSocket channel set. If it does, the WebSocket channel used to initiate the command transmission is identified as the target WebSocket channel, thus determining the object to which this Event is sent. Finally, the Event is sent to the corresponding browser through the target WebSocket channel, allowing the context dialogue to be displayed in the browser's text dialog box. This enables users to directly see the context dialogue corresponding to the spoken words in real time, improving the efficiency of NLP testing.

[0039] The second aspect of this application provides a real-time NLP testing device based on FreeSwitch and WebRTC technologies, specifically including:

[0040] The account registration module is used to receive a preset account transmitted from the browser via WebRTC through FreeSWITCH and obtain the valid user account corresponding to the preset account.

[0041] The channel establishment module is used to receive the start command sent by the browser and establish a bidirectional communication channel between the FreeSWITCH and the browser based on the valid user account.

[0042] The dialogue acquisition module is used to acquire text information corresponding to the voice stream transmitted through the bidirectional communication channel via the freeswitch based on the bidirectional communication channel, and to parse the text information through NLP to obtain the context dialogue corresponding to the text information.

[0043] The dialogue display module is used to send the context dialogue to the browser via the freeswitch, so that the browser displays the context dialogue in a text dialog box.

[0044] By adopting the above technical solution, the account registration module receives the preset account transmitted by WebRTC in the browser through FreeSWITCH. After registration and authentication, the corresponding valid user account is obtained. The channel establishment module receives the start command sent by the browser and establishes a two-way communication channel between FreeSWITCH and the browser based on the valid user account. Then, the dialogue acquisition module obtains the text information corresponding to the transmitted voice stream through FreeSWITCH based on the two-way communication channel, and parses the text information through NLP to obtain the corresponding context dialogue. Finally, the dialogue display module sends the context dialogue to the browser for display, thereby improving the efficiency of NLP testing.

[0045] In summary, this application includes at least one of the following beneficial technical effects:

[0046] The browser's WebRTC sends a preset account to FreeSwitch on the server. FreeSwitch registers and authenticates the preset account to obtain a valid user account. Then, FreeSwitch receives a start command from the browser, bridges the valid user account existing in the context, and establishes communication between FreeSwitch and the valid user account, forming a bidirectional communication channel between FreeSwitch and the browser. The audio stream transmitted between the browser and FreeSwitch via this bidirectional communication channel is acquired and converted into corresponding text information by FreeSwitch. Finally, FreeSwitch bridges the communication channel with NLP, sends the text information to NLP for parsing to obtain the context dialogue corresponding to the text information, and sends it to the browser through the bidirectional communication channel to be displayed in the dialog text box on the browser page. This bidirectional communication channel enables real-time testing of NLP through the transmission of media streams (audio and text), thus eliminating the need for extensive testing and verification of NLP via PSTN, significantly improving NLP testing efficiency. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the architecture of a real-time NLP testing system based on FreeSwitch and WebRTC technologies provided in an embodiment of this application;

[0048] Figure 2 This is a flowchart illustrating a real-time NLP testing method based on FreeSwitch and WebRTC technologies provided in an embodiment of this application.

[0049] Figure 3 This is a flowchart illustrating another method for real-time NLP testing based on FreeSwitch and WebRTC technologies provided in this application embodiment;

[0050] Figure 4 This is a schematic diagram of the structure of a real-time NLP testing device based on FreeSwitch and WebRTC technology provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of another NLP real-time testing device based on FreeSwitch and WebRTC technology provided in this application embodiment.

[0052] Explanation of the attached diagram labels: 11. Account registration module; 12. Channel establishment module; 13. Dialogue acquisition module; 14. Dialogue display module. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0054] In the description of the embodiments in this application, words such as "illustrative," "for example," or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "illustrative," "for example," or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "illustrative," "for example," or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, or A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0056] See Figure 1 This application discloses an architecture diagram of a real-time NLP testing system based on FreeSwitch and WebRTC technologies. Specifically, it includes a terminal and a server. The terminal can connect directly or indirectly to the server via wired or wireless networks. The terminal can be an electronic device such as a mobile phone, tablet computer, e-book reader, multimedia playback device, wearable device, or PC (Personal Computer). The server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0057] The terminal's built-in browser establishes a bidirectional communication channel with the FreeSWITCH on the server, namely a WebRTC channel and a WebSocket channel. Based on this bidirectional communication channel, the FreeSWITCH on the server obtains the dialogue audio (audio stream) for voice testing from the WebRTC channel, converts the audio into text, sends the text to the NLP on the server for parsing to obtain the corresponding context dialogue, then the NLP sends the context dialogue back to the FreeSWITCH, and finally the FreeSWITCH sends the context dialogue back to the terminal via the WebSocket channel, thus displaying the context dialogue in the browser's dialog text box.

[0058] See Figure 2 This application discloses a flowchart illustrating a real-time NLP testing method based on FreeSwitch and WebRTC technologies. This method can be implemented using a computer program or run on a von Neumann architecture-based NLP real-time testing device using FreeSwitch and WebRTC technologies. The computer program can be integrated into an application or run as a standalone utility application, specifically including:

[0059] S101: Receive the preset account transmitted by WebRTC in the browser via FreeSWITCH, and obtain the valid user account corresponding to the preset account.

[0060] In one feasible implementation, a preset account transmitted via WebRTC in a browser is received through FreeSwitch based on the SIP protocol.

[0061] Register the preset account using FreeSWITCH to obtain the registered account;

[0062] FreeSwitch performs 401 authentication on the registered account. If the 401 authentication is successful, the registered account is confirmed as a valid user account.

[0063] Specifically, FreeSwitch is a powerful softswitch system in the CIT field, running on a server and providing voice, video, and chat functions. FreeSwitch can be used as a switch engine, PBX, multimedia gateway, and multimedia server. Web Real-Time Communications (WebRTC) is a real-time communication technology that allows web applications or websites to establish peer-to-peer connections between browsers without the need for an intermediary, enabling the transmission of video and / or audio streams or other arbitrary data. In this embodiment, the browser integrates WebRTC. Users send a preset account to FreeSwitch on the server through browser operations, specifically via WebRTC based on the SIP protocol. After receiving the preset account transmitted via WebRTC, FreeSwitch registers the preset account and performs 401 authentication, ensuring that only visitors logged in with the preset account are allowed to access the application. After successful authentication, a valid user account corresponding to the preset account is obtained, which is a valid online user that can communicate, making it convenient for the valid user account to establish communication with FreeSwitch in the future.

[0064] S102: Receives the start command sent by the browser and establishes a two-way communication channel between FreeSWITCH and the browser based on a valid user account.

[0065] Specifically, the start command refers to the start command for Natural Language Processing (NLP) testing. NLP is a discipline that studies language problems in human-computer interaction. Based on the varying levels of technical difficulty, these systems can be categorized into three types: simple matching, fuzzy matching, and paragraph understanding. Furthermore, NLP has become the foundation for creating chatbots.

[0066] Users send a start command to the server by clicking the start button on the browser page, notifying the server that an NLP test is required. After receiving the start command from the browser, the FreeSWITCH on the server responds by sending a command to the browser allowing the test (allowing voice calls). Finally, FreeSWITCH receives the official call command from the browser again and simultaneously bridges the authenticated and verified user account, establishing communication between the valid user account and the FreeSWITCH on the server, thus establishing a bidirectional communication channel between FreeSWITCH and the browser corresponding to the valid user account. Bridging refers to the process of forwarding network data packets based on the link layer address in the OSI network model; simply put, bridging connects two different physical LANs, establishing bidirectional communication between them.

[0067] It should be noted that the bidirectional communication channels are the WebRTC channel and the WebSocket channel, specifically the WebRTC transmission channel and the WebSocket transmission channel. The WebRTC channel is the specific multimedia data transmission channel relied upon for audio and video communication between WebRTC and FreeSwitch in the server. WebSocket is a protocol for full-duplex communication over a single TCP connection. WebSocket simplifies data exchange between the browser and server, allowing the server to proactively push data to the browser. Through the WebSocket API, the browser and server only need to complete one HTTP handshake to establish a persistent connection, enabling real-time bidirectional data transmission through the corresponding WebSocket channel. Furthermore, the browser and server establish a long-lived connection through the WebSocket channel, avoiding the need for HTTP requests for each data transmission as with short connections, thus conserving resources and simplifying data transmission. This is existing technology and will not be elaborated further. It should also be noted that WebSocket is a built-in object in the browser.

[0068] S103: Based on the bidirectional communication channel, obtain the text information corresponding to the voice stream transmitted through the bidirectional communication channel via FreeSwitch, and parse the text information using NLP to obtain the context dialogue corresponding to the text information.

[0069] Specifically, after the bidirectional communication channel between FreeSwitch and the browser is established, the server obtains the audio stream from the WebRTC channel through FreeSwitch. This audio stream is the dialogue between the browser and FreeSwitch used for NLP testing. All voice conversations between the user and FreeSwitch through the browser are transmitted via the WebRTC channel. After obtaining the audio stream, FreeSwitch converts it into corresponding text information. This can be done using a pre-built speech recognition tool. Next, FreeSwitch establishes communication with the NLP and encapsulates the text information. The encapsulated text information is then sent to the NLP, which parses it to determine the contextual logic and ultimately obtains the contextual dialogue corresponding to the text information.

[0070] It's important to note that the text information obtained by FreeSwitch from the speech stream is a cohesive dialogue text without any contextual logic. Therefore, to allow users to view text that matches the spoken dialogue (with consistent contextual logic) on the browser's front-end page, enabling more efficient NLP testing, it's necessary to use NLP to parse the contextual logic of the text information and segment it according to that logic to ultimately obtain the corresponding contextual dialogue.

[0071] S104: Send the context dialog to the browser via FreeSwitch so that the browser displays the context dialog in a text dialog box.

[0072] Specifically, after obtaining the context dialogue by parsing the text information through NLP, a request to obtain the context dialogue is sent to NLP through FreeSWITCH. FreeSWITCH then obtains the context dialogue from NLP and sends it to the browser corresponding to the valid user account through a two-way communication channel. This allows the context dialogue to be displayed in the browser's text dialog box in real time, enabling users to observe the NLP test results in real time, thereby improving the efficiency of NLP testing.

[0073] See Figure 3 This application discloses a flowchart illustrating another method for real-time NLP testing based on FreeSwitch and WebRTC technologies. This method can be implemented using a computer program or run on a von Neumann architecture-based NLP real-time testing device using FreeSwitch and WebRTC technologies. The computer program can be integrated into an application or run as a standalone utility application, specifically including:

[0074] S201: Receive the preset account transmitted in the browser via WebRTC through FreeSWITCH and obtain the valid user account corresponding to the preset account.

[0075] For details, please refer to step S101, which will not be repeated here.

[0076] S202: Receive the start command sent by the browser via WebSocket, and FreeSWITCH initiates a media call request to the browser.

[0077] Specifically, after identifying a valid user account within the context, the user logs into the NLP real-time testing system using this valid user account. This causes the browser's front-end page to display a "Start Test" button for voice testing. In other embodiments, the browser may display a "Start Test" button for text testing, or both voice and text testing buttons simultaneously. After the user clicks the "Start Test" button, the browser sends a start command to FreeSWITCH on the server via WebSocket. For FreeSWITCH, upon receiving the string corresponding to the start command, it first parses the encapsulation parameters of the string to determine that the browser sent a start command for voice testing. Then, FreeSWITCH proactively initiates a media call request to the browser via WebSocket, thus preparing for the formal start of the voice dialogue and NLP testing. For example, after receiving the start command from the browser, FreeSWITCH generates and sends the voice message "Test can begin" to the browser.

[0078] S203: After initiating a media call request to the browser through FreeSWITCH, it receives the call instructions sent by the browser and bridges the valid user account. After negotiating with SIP signaling, it establishes a bidirectional communication channel between FreeSWITCH and the browser.

[0079] Specifically, the Session Initialization Protocol (SIP) is a multimedia communication protocol developed by the Internet Engineering Task Force (IETF). It is a text-based application-layer control protocol used to establish, modify, and terminate multimedia sessions. SIP itself does not provide services; it serves as a component of the multimedia architecture along with other protocols. After a media call request is initiated to the browser via FreeSWITCH, the browser sends a call command to FreeSWITCH via WebSocket. Upon receiving the call command, FreeSWITCH initiates the voice dialogue between the browser and FreeSWITCH. In other embodiments, after a media call request is initiated to the browser via FreeSWITCH, the voice dialogue between the browser and FreeSWITCH officially begins, and the user can send voice messages to FreeSWITCH on the server through the browser. It should be noted that the browser's corresponding terminal is equipped with hardware devices (such as a sound card) that support human-computer voice interaction, making NLP testing more convenient and effective. The terminal can be an electronic device such as a mobile phone, tablet computer, or PC (Personal Computer).

[0080] Next, a valid user account existing in the context is bridged via FreeSWITCH, and SIP signaling negotiation is performed simultaneously. After the SIP signaling negotiation is completed, voice messages between the browser and FreeSWITCH can be converted into multimedia messages with a unified SIP signaling format, thereby establishing bidirectional communication between the valid user account and the FreeSWITCH on the server, that is, establishing a bidirectional communication channel between FreeSWITCH and the browser corresponding to the valid user account. This facilitates the subsequent transmission of multimedia messages through the bidirectional communication channel. This is existing technology and will not be elaborated further here.

[0081] S204: Obtain the RTP voice stream transmitted through the WebRTC channel via FreeSWITCH.

[0082] S205: Convert the RTP voice stream into corresponding text information using ASR technology.

[0083] Specifically, after the bidirectional communication channel is established, since the voice stream transmission within the WebRTC channel is based on RTP / RTCP media stream transmission, the voice stream transmitted between the browser and FreeSwitch via the WebRTC channel is an RTP voice stream. The Real-time Transport Protocol (RTP) is a network transport protocol that details the standard data packet format for transmitting audio over the Internet. The Real-time Transport Control Protocol (RTCP) is a functional part of RTP as a transport protocol and involves flow control and congestion control for other transport protocols. Furthermore, RTP / RTCP is also a fundamental requirement for creating SIP multimedia sessions.

[0084] This involves using FreeSWITCH to capture RTP voice streams transmitted over the WebRTC channel, essentially capturing the conversational audio. The method involves using a pre-defined media bug to help FreeSWITCH extract the conversational audio from the WebRTC channel. The media bug is primarily used for capturing and extracting call audio data.

[0085] After acquiring the RTP voice stream, FreeSwitch is used to perform speech recognition and convert the RTP voice stream into text information, primarily through Automatic Speech Recognition (ASR) technology. ASR is a technology that converts human speech into text. It should be noted that the converted text information is in XML format.

[0086] In one possible implementation, after step S205, the method further includes: the FreeSwitch sending the RTP audio stream to the browser via the WebRTC channel, so that the browser can play the RTP audio stream; or

[0087] FreeSWITCH sends text messages to the browser via a WebSocket channel, allowing the browser to play the text messages. This enables users to hear the spoken dialogue in real time and / or view the text dialogue in real time on their browser.

[0088] S206: Encapsulate the text information to obtain the encapsulated text information.

[0089] S207: Send the encapsulated text information to NLP via FreeSwitch for parsing to obtain the context dialogue corresponding to the text information.

[0090] Specifically, FreeSWITCH obtains the text information corresponding to the RTP voice stream, and then encapsulates the XML-formatted text information into an object. The encapsulation method uses Java and access modifiers (private, default, protected, public) to encapsulate the text information. This encapsulation serves two purposes: first, it ensures the security of the text information; second, it facilitates external calls via an interface.

[0091] After obtaining the encapsulated text information, the communication channel with NLP is bridged through FreeSwitch, enabling FreeSwitch to establish communication with NLP. Then, FreeSwitch sends the encapsulated text information to NLP, which then parses and performs semantic analysis on the encapsulated text information to further determine the context logic of the text information, segments the text information, and finally obtains the context dialogue.

[0092] For example, when a user conducts a voice test through a browser, the dialogue with FreeSwitch on the server consists of the following audio: "Hello, it's nice to meet you," "Hello, it's nice to meet you too," "How have you been lately?" "I'm doing well, how about you?" The corresponding text information would be: "Hello, it's nice to meet you. Hello, it's nice to meet you too. How have you been lately? I'm doing well, how about you?" This text information lacks any contextual logic. Therefore, through NLP parsing and analysis, the corresponding contextual dialogue can be obtained, namely:

[0093] Hello, it's nice to meet you.

[0094] Hello, it's nice to meet you too.

[0095] "how have you been"

[0096] "I'm doing well lately, how about you?"

[0097] S208: The context dialogue is obtained from NLP by FreeSwitch through the NLP interface.

[0098] S209: Encapsulate the context dialogue to obtain the Event.

[0099] S210: Based on the MQ message mechanism, it sends Events to the browser through a WebSocket channel, so that the browser can display the context dialog in a text dialog box.

[0100] In one feasible approach, based on the MQ message mechanism, the set of WebSocket channels recorded by the server is obtained through FreeSWITCH, and the target WebSocket channel for starting command transmission is selected from the set of WebSocket channels.

[0101] Events are sent to the browser via the target WebSocket channel, causing the browser to display the contextual dialogue in a text dialog box.

[0102] Specifically, after obtaining the context dialogue corresponding to the text information through NLP, the NLP interface is called via FreeSwitch to retrieve the context dialogue from NLP. Then, the context dialogue is encapsulated into an Event. An Event is a type of encapsulated delegate that includes an event issuer and an event subscriber. The event issuer and subscriber use delegates to trigger events on the sender and receiver sides. In this embodiment, the Event is the event of the recognized text (forming the context dialogue) subscribed to by the browser through a WebSocket channel.

[0103] After receiving an Event, FreeSwitch uses a Message Queue (MQ) mechanism to place the Event to be sent to the browser into the message queue. MQ is a "First-In, First-Out" (FIFO) data structure. It involves placing messages to be transmitted in a queue, using a queue mechanism to implement message delivery—the producer generates messages and puts them into the queue, then the user processes them. Users can specify a queue to pull messages or subscribe to a corresponding queue, where the MQ server pushes messages to them. If multiple Events exist in the message queue, meaning multiple users are using browsers for voice testing, each browser needs to receive the corresponding Event sent by FreeSwitch.

[0104] Furthermore, the server retrieves the WebSocket channel set obtained by the Java application, as Java applications automatically record all WebSocket channels established through the browser. Next, it checks if the WebSocket channel used for the initial command transmission exists in the WebSocket channel set. If it does, this WebSocket channel is identified as the target WebSocket channel, thus determining the object to which the browser's subscribed Event should be sent. Because many browsers are involved, it is necessary to determine the correct object to which the Event should be sent.

[0105] After identifying the target WebSocket channel, FreeSWITCH sends the Event through that channel to the corresponding browser. The browser then displays the context dialogue in a text dialog box. This ensures that the context dialogue corresponding to the Event matches the voice test conducted through that browser, avoiding sending to the wrong recipient.

[0106] The implementation principle of this application's embodiment of the NLP real-time testing method based on FreeSWITCH and WebRTC technology is as follows: WebRTC, integrated in the browser, sends a preset account to FreeSWITCH on the server. FreeSWITCH registers and authenticates the preset account, making it a valid user account. Then, FreeSWITCH receives a start command from the browser, bridges the valid user account existing in the context, and establishes communication between FreeSWITCH and the valid user account. This establishes bidirectional communication between FreeSWITCH and the browser, forming a bidirectional communication channel. The audio stream transmitted between the browser and FreeSWITCH through this bidirectional communication channel is acquired, and FreeSWITCH converts the audio stream into corresponding text information. Finally, FreeSWITCH bridges the communication channel with NLP, sends the text information to NLP for parsing to obtain the context dialogue corresponding to the text information, and sends it to the browser through the bidirectional communication channel, displaying the dialogue text box on the browser page. Thus, by transmitting media streams (audio stream and text) through the bidirectional communication channel, NLP can be tested in real time, eliminating the need for extensive PSTN testing to verify NLP and significantly improving NLP testing efficiency.

[0107] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0108] Please see Figure 4 This is a schematic diagram of the structure of a real-time NLP testing device based on FreeSwitch and WebRTC technology provided in this application embodiment. This device can be implemented as all or part of a whole through software, hardware, or a combination of both. The device 1 includes an account registration module 11, a channel establishment module 12, a dialogue acquisition module 13, and a dialogue display module 14.

[0109] Account registration module 11 is used to receive preset accounts transmitted from the browser via WebRTC through FreeSWITCH and obtain the valid user accounts corresponding to the preset accounts;

[0110] Channel establishment module 12 is used to receive the start command sent by the browser and establish a bidirectional communication channel between FreeSWITCH and the browser based on a valid user account;

[0111] The dialogue acquisition module 13 is used to acquire the text information corresponding to the voice stream transmitted through the bidirectional communication channel via FreeSwitch, and to parse the text information through NLP to obtain the context dialogue corresponding to the text information.

[0112] Dialogue display module 14 is used to send the context dialogue to the browser via FreeSwitch, so that the browser can display the context dialogue in a text dialog box.

[0113] Optional, account registration module 11, specifically used for:

[0114] Based on the SIP protocol, a preset account is received from WebRTC transmission in the browser via FreeSWITCH;

[0115] Register a pre-defined account using FreeSWITCH to obtain the registered account;

[0116] FreeSwitch performs 401 authentication on the registered account. If the 401 authentication is successful, the registered account is confirmed as a valid user account.

[0117] Optional, channel establishment module 12, specifically used for:

[0118] The browser receives the start command via WebSocket, and FreeSWITCH initiates a media call request to the browser.

[0119] After initiating a media call request to the browser via FreeSWITCH, it receives the call instructions sent by the browser and bridges a valid user account. After negotiating with SIP signaling, it establishes a bidirectional communication channel between FreeSWITCH and the browser.

[0120] Optional, the dialogue acquisition module 13 is specifically used for:

[0121] Obtain RTP voice streams transmitted via the WebRTC channel using FreeSWITCH;

[0122] Convert RTP audio streams into corresponding text information using ASR technology;

[0123] The text information is encapsulated to obtain the encapsulated text information;

[0124] The encapsulated text information is sent to NLP for parsing via FreeSWITCH to obtain the context dialogue corresponding to the text information.

[0125] Optional, such as Figure 5 As shown, device 1 also includes a text playback module 15, specifically used for:

[0126] FreeSwitch sends an RTP audio stream to the browser via a WebRTC channel, allowing the browser to play the RTP audio stream; or

[0127] FreeSWITCH sends text messages to the browser via a WebSocket channel, allowing the browser to play the text messages.

[0128] Optional, the dialogue display module 14 is specifically used for:

[0129] FreeSwitch obtains the context dialogue from NLP through the NLP interface;

[0130] The context dialogue is encapsulated to obtain the Event;

[0131] Based on the MQ message mechanism, Events are sent to the browser via a WebSocket channel, so that the browser can display the context dialogue in a text dialog box.

[0132] Optionally, the dialogue display module 14 is also used for:

[0133] Based on the MQ message mechanism, the set of WebSocket channels recorded by the server is obtained through FreeSWITCH, and the target WebSocket channel for starting command transmission is selected from the set of WebSocket channels.

[0134] Events are sent to the browser via the target WebSocket channel, so that the browser displays the context dialog in a text dialog box.

[0135] It should be noted that the above embodiments of the NLP real-time testing device based on FreeSWITCH and WebRTC technology, when executing the NLP real-time testing method based on FreeSWITCH and WebRTC technology, are only illustrative examples of the above-described functional module divisions. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the above embodiments of the NLP real-time testing device based on FreeSWITCH and WebRTC technology and the NLP real-time testing method based on FreeSWITCH and WebRTC technology belong to the same concept, and their implementation process is detailed in the method embodiments, which will not be repeated here.

[0136] This application also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it employs a real-time NLP testing method based on FreeSwitch and WebRTC technology as described in the above embodiments.

[0137] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0138] The above-described method for real-time NLP testing based on FreeSwitch and WebRTC technology is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above method.

[0139] This application also discloses an electronic device in which a computer program is stored in a computer-readable storage medium. When the computer program is loaded and executed by a processor, it employs the aforementioned method for real-time NLP testing based on FreeSwitch and WebRTC technologies.

[0140] The electronic device can be a desktop computer, a laptop computer, or a cloud server, and the electronic device includes, but is not limited to, a processor and a memory. For example, the electronic device may also include input / output devices, network access devices, and buses.

[0141] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0142] The memory can be an internal storage unit of an electronic device, such as a hard disk or RAM, or an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the electronic device. Furthermore, the memory can be a combination of an internal storage unit and an external storage device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0143] In this electronic device, the NLP real-time testing method based on FreeSwitch and WebRTC technology described in the above embodiment is stored in the memory of the electronic device and loaded and executed on the processor of the electronic device for convenient use.

[0144] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for NLP real-time testing based on freeswitch and WebRTC technology, characterized in that, Applied to a server, the method includes: The system receives a preset account transmitted via WebRTC in the browser through FreeSWITCH and obtains the valid user account corresponding to the preset account. Receive a start command sent by the browser, and establish a bidirectional communication channel between the FreeSWITCH and the browser based on the valid user account, including: receiving the start command sent by the browser via WebSocket, and the FreeSWITCH initiating a media call request to the browser; After initiating a media call request to the browser through the FreeSWITCH, the FreeSWITCH receives the call instruction sent by the browser and bridges the valid user account. A bidirectional communication channel is established between the FreeSWITCH and the browser through SIP signaling negotiation. The bidirectional communication channel is the WebRTC channel and the WebSocket channel. Based on the bidirectional communication channel, the FreeSwitch obtains the audio stream from the WebRTC channel, converts the audio stream into text information, and parses the text information using NLP to obtain the context dialogue corresponding to the text information; The context dialog is sent to the browser via a WebSocket channel using FreeSWITCH, so that the browser displays the context dialog in a text dialog box.

2. The method for NLP real-time test based on freeswitch and WebRTC technology according to claim 1, characterized in that, The step of receiving a preset account transmitted via WebRTC in a browser through FreeSWITCH and obtaining the valid user account corresponding to the preset account includes: Based on the SIP protocol, a preset account is received from WebRTC transmission in the browser via FreeSWITCH; Register the preset account through FreeSWITCH to obtain the registered account; The registered account is authenticated using FreeSwitch. If the authentication is successful, the registered account is confirmed as a valid user account.

3. The method for real-time NLP testing based on FreeSwitch and WebRTC technology according to claim 1, characterized in that, Based on the bidirectional communication channel, the FreeSwitch obtains the audio stream from the WebRTC channel, converts the audio stream into text information, and parses the text information using NLP to obtain the context dialogue corresponding to the text information, including: The RTP voice stream transmitted through the WebRTC channel is obtained through the freeswitch; The RTP voice stream is converted into corresponding text information using ASR technology; The text information is encapsulated to obtain encapsulated text information; The encapsulated text information is sent to NLP for parsing via FreeSWITCH to obtain the context dialogue corresponding to the text information.

4. The method for real-time NLP testing based on FreeSwitch and WebRTC technology according to claim 3, characterized in that, After converting the RTP voice stream into corresponding text information using ASR technology, the process further includes: The FreeSwitch program sends the RTP audio stream to the browser via the WebRTC channel, allowing the browser to play the RTP audio stream; or The FreeSWITCH sends the text information to the browser via the WebSocket channel, so that the browser can play the text information.

5. The method for real-time NLP testing based on FreeSwitch and WebRTC technology according to claim 1, characterized in that, The step of sending the context dialog to the browser via the FreeSWITCH-based WebSocket channel, so that the browser displays the context dialog in a text dialog box, includes: The context dialogue is obtained by the FreeSwitch from the NLP through the NLP interface; The context dialogue is encapsulated to obtain an Event; Based on the MQ message mechanism, the Event is sent to the browser through the WebSocket channel, so that the browser displays the context dialog in a text dialog box.

6. The method for real-time NLP testing based on FreeSwitch and WebRTC technology according to claim 5, characterized in that, The method of sending the Event to the browser via the WebSocket channel based on the MQ message mechanism, so that the browser displays the context dialog in a text dialog box, includes: Based on the MQ message mechanism, the set of WebSocket channels recorded by the server is obtained through the FreeSWITCH, and the target WebSocket channel for the start command transmission is selected from the set of WebSocket channels. The Event is sent to the browser via the target WebSocket channel, so that the browser displays the context dialog in a text dialog box.

7. A real-time NLP testing device based on FreeSWITCH and WebRTC technology, used to implement the real-time NLP testing method based on FreeSWITCH and WebRTC technology as described in any one of claims 1 to 6, characterized in that, include: The account registration module is used to receive a preset account transmitted from the browser via WebRTC through FreeSWITCH and obtain the valid user account corresponding to the preset account. The channel establishment module is used to receive the start command sent by the browser and establish a bidirectional communication channel between the FreeSWITCH and the browser based on the valid user account. This includes receiving the start command sent by the browser via WebSocket and the FreeSWITCH initiating a media call request to the browser. After initiating a media call request to the browser through the FreeSWITCH, the FreeSWITCH receives the call instruction sent by the browser and bridges the valid user account. A bidirectional communication channel is established between the FreeSWITCH and the browser through SIP signaling negotiation. The bidirectional communication channel is the WebRTC channel and the WebSocket channel. The dialogue acquisition module is used to acquire the voice stream from the WebRTC channel based on the bidirectional communication channel, convert the voice stream into text information, and parse the text information through NLP to obtain the context dialogue corresponding to the text information. The dialog display module is used to send the context dialog to the browser via the FreeSWITCH based on the WebSocket channel, so that the browser displays the context dialog in a text dialog box.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the method described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor loads and executes the computer program, it employs the method described in any one of claims 1-6.

Images