Voice processing method and system based on full-duplex communication protocol, computer device

Abstract

The application provides a voice processing method and system based on a full-duplex communication protocol, and a computer device. The voice processing method comprises the following steps: receiving first audio information of a user and converting the first audio information into a plurality of first text information; performing a punctuation service on the first text information to obtain second text information; transmitting the plurality of second text information in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information comprises a current round number and a request service delay time; determining a first waiting time for replying to the user according to the punctuation information and the request service delay time; matching the second text information with a preset intent library to obtain reply information of the current round; comparing the current round number with a preset reply round number to obtain a comparison result, and replying to the user according to the comparison result, the reply information and the first waiting time, or canceling the reply to the user according to the comparison result and the reply information. The application can effectively improve the dialogue experience of the user and further improve the user satisfaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a voice processing method and system based on a full-duplex communication protocol, and a computer device. BACKGROUND

[0002] In today's industry and service industry, providing a question and answer service to users is an indispensable part of production and consumption. With the continuous development of artificial intelligence, voice recognition technology is becoming more and more mature. Using an intelligent voice customer service robot is an effective choice to save manpower. By searching for some problems, the user's desired answer can be quickly and accurately given. However, the voice interaction customer service system of the related technology usually has the customer service robot reply to the user after speaking a complete sentence. This causes the customer service system to be unable to effectively handle the problem of being interrupted by the user or interrupting the user in voice dialogue interaction, resulting in a poor dialogue experience for the user. Moreover, the length of the pause in the voice dialogue is also uncontrollable, and there are often cases of too fast answering the user's question or too slow replying, which makes the user and the customer service robot have poor communication in the voice dialogue, reducing the user's satisfaction. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a voice processing method and system based on a full-duplex communication protocol, and a computer device, which can effectively improve the user's dialogue experience and thus improve the user's satisfaction.

[0004] In a first aspect, an embodiment of the present application provides a voice processing method based on a full-duplex communication protocol, comprising:

[0005] receiving first audio information of a user, and converting the first audio information into a plurality of first text information;

[0006] punctuating the first text information to obtain second text information with punctuation;

[0007] transmitting a plurality of the second text information in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time;

[0008] determining a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time;

[0009] matching the second text information with a preset intent library to obtain reply information of the current round;

[0010] The current round number and the preset reply round number are compared to obtain a comparison result, and a user is replied according to the comparison result, the reply information and the first waiting time, or the user is cancelled according to the comparison result and the reply information.

[0011] In some embodiments, the converting the first audio information into a plurality of first text information comprises:

[0012] The first audio information is converted into binary data.

[0013] Voice features are extracted from the binary data and input into a preset ASR speech recognition model to obtain a plurality of first text information.

[0014] In some embodiments, the punctuating the first text information to obtain punctuated second text information comprises:

[0015] The first text information is encoded by a preset BERT pre-training model.

[0016] A BiLSTM+CRF algorithm is used to predict the punctuation corresponding to the first text information to obtain second text information, wherein the second text information is a complete sentence with punctuation.

[0017] In some embodiments, the determining a first waiting time for replying to a user in a dialogue process according to the punctuation information and the request service delay time comprises:

[0018] The state information of the user in the current dialogue process is determined according to the punctuation information.

[0019] A second waiting time is determined according to the state information.

[0020] The difference between the second waiting time and the request service delay time is calculated to obtain the first waiting time.

[0021] In some embodiments, the matching the second text information with a preset intent library to obtain reply information of the current round comprises:

[0022] The second text information is subjected to speech recognition to obtain a semantic recognition result, wherein the semantic recognition result includes user intent information and entity information.

[0023] The intent information and the entity information are matched with the intent library to obtain reply information of the current round.

[0024] In some embodiments, the replying to the user according to the comparison result, the reply information and the first waiting time, or cancelling the reply to the user according to the comparison result and the reply information comprises:

[0025] If the comparison result is that the current round number is greater than the preset reply round number, it is determined that the reply information of the current round is invalid, and the reply user is cancelled.

[0026] If the comparison result is that the current round number is less than or equal to the preset reply round number, the reply information is converted into second audio information, and the second audio information is output to the user after a first waiting time.

[0027] In some embodiments, after obtaining the reply information of the current round, if third audio information is received from the user, it is determined that the reply information of the current round is invalid, and the reply user is cancelled.

[0028] In a second aspect, the embodiments of the present application also provide a voice processing system based on a full-duplex communication protocol, comprising:

[0029] An acquisition module is configured to receive first audio information of a user, and convert the first audio information into a plurality of first text information.

[0030] A punctuation marking module is configured to perform punctuation marking service on the first text information to obtain second text information with punctuation.

[0031] A multi-round transmission module is configured to transmit a plurality of the second text information in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time.

[0032] A time determination module is configured to determine a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time.

[0033] An information matching module is configured to match the second text information with a preset intent library to obtain reply information of the current round.

[0034] A comparison module is configured to compare the current round number with a preset reply round number to obtain a comparison result, and reply to the user according to the comparison result, the reply information and the first waiting time, or cancel the reply to the user according to the comparison result and the reply information.

[0035] In a third aspect, the embodiments of the present application also provide a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the voice processing method of the first aspect when executing the computer program.

[0036] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium storing computer executable instructions for executing the voice processing method according to the first aspect.

[0037] The embodiments of the present application include a voice processing method and system based on a full-duplex communication protocol, a computer device, and a computer readable storage medium. The voice processing method includes receiving first audio information of a user, converting the first audio information into a plurality of first text information, performing a punctuation service on the first text information to obtain second text information with punctuation, transmitting the plurality of second text information in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time, determining a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time, matching the second text information with a preset intent library to obtain reply information of the current round, comparing the current round number with a preset reply round number to obtain a comparison result, and replying to the user according to the comparison result, the reply information, and the first waiting time, or canceling the reply to the user according to the comparison result and the reply information. According to the technical solution of the embodiments of the present application, the first audio information input by the user is converted into a plurality of first text information for analysis, and the first text information is subjected to a punctuation service to obtain second text information, which can lay a foundation for subsequent understanding of the user's voice. Based on the full-duplex communication protocol, the second text information is transmitted in multiple rounds, which can improve the speed of responding to the user in the dialogue process. Since the second text information carries the key information and punctuation information of the current round, the first waiting time for replying to the user in the dialogue process is determined by considering the punctuation information and the request service delay time, which realizes personalized calculation of the speaking time of the customer service robot, effectively improves the dialogue experience of the user, and further improves the user satisfaction. In addition, the intent library is used to match the second text information, which can quickly obtain the reply information of the current round. By further comparing the current round number with the preset reply round number, it is determined whether to reply to the user after the first time or to cancel the reply to the user according to the comparison result, so that the user's question in the dialogue process can be replied to more intelligently and efficiently.

[0038] Additional features and advantages of the application will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are used to provide further understanding of the technical solutions of the present application, and constitute a part of the specification, and are used to explain the technical solutions of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0040] Figure 1 is a flowchart of a voice processing method provided by an embodiment of the present application;

[0041] Figure 2 is a structural schematic diagram of a voice processing system provided by an embodiment of the present application;

[0042] Figure 3 is a structural schematic diagram of a voice processing system provided by an embodiment of the present application;

[0043] Figure 4 is a flowchart of a voice processing method provided by another embodiment of the present application;

[0044] Figure 5 is a flowchart of a voice processing method provided by another embodiment of the present application;

[0045] Figure 6 is a flowchart of a voice processing method provided by another embodiment of the present application;

[0046] Figure 7 is a flowchart of a voice processing method provided by another embodiment of the present application;

[0047] Figure 8 is a flowchart of a voice processing method provided by another embodiment of the present application;

[0048] Figure 9 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0050] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the description, claims, or above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. In addition, the terms "include" or "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or devices. The term "and / or" used herein is only a description of the same field of associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0051] The voice interaction customer service system in the related art still mostly stays in single work single channel interaction, so that the customer service robot usually replies to the user's question after speaking a complete script, which causes the customer service system to be unable to effectively handle the user's interruption or interruption of the user's question in voice dialogue interaction, resulting in poor dialogue experience for the user. Moreover, the pause length in voice dialogue also becomes uncontrollable, and the situation of too fast answering the user's question or too slow reply often occurs. Too fast answering the user's question will make the user feel very abrupt, and too slow reply will also make the user feel impatient and think that the customer service robot is not intelligent, so that the user and the customer service robot communicate poorly in voice dialogue, reducing the user's satisfaction.

[0052] The embodiments of the present application will be further described below with reference to the accompanying drawings.

[0053] As shown in Figure 1 and Figure 2 , the first aspect embodiment of the present application provides a voice processing method based on a full-duplex communication protocol, which is applied to a voice processing system 200 in Figure 2 , and a full-duplex communication protocol is used for data transmission in the voice processing system 200.

[0054] As shown in Figure 1 , the voice processing method includes but is not limited to steps S110 to S160:

[0055] Step S110: receiving first audio information of a user, and converting the first audio information into a plurality of first text information;

[0056] AsFigure 3 As shown, the voice processing system 200 interacts with the user through the audio collection system 300 and the client 400. The user transmits first audio information to the audio collection system 300 through the client 400 (such as a mobile phone APP, a webpage, etc.) for storage. The audio collection system 300 transmits the first audio information to the voice processing system 200. The voice processing system 200 converts the first audio information into a plurality of first text information by calling a voice-to-text service. The audio data is converted into text data, which facilitates data processing in the voice interaction process.

[0057] Step S120: punctuate the first text information to obtain punctuated second text information;

[0058] The first text information is punctuated to add punctuation to the first text information, and the obtained second text information is a complete sentence with punctuation, which lays a foundation for better understanding of the user's voice.

[0059] Step S130: transmit the plurality of second text information in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time;

[0060] It should be noted that the HTTP / 2 protocol is used for data transmission in the voice processing system 200. When receiving a user's dialogue request, the second text information can be transmitted in multiple rounds without waiting for the user to finish a complete sentence, which is beneficial to improve the user's dialogue experience.

[0061] It should be noted that the plurality of second text information is transmitted in multiple rounds, and the second text information carries key information and punctuation information of the current round. The key information includes a current round number and a request service delay time. The current round number is an agreed integer key. The plurality of second text information is transmitted in multiple rounds, and the current round number is increased by 1 in each round of transmission. The request service delay time includes the delay time generated by calling the voice-to-text service, the punctuation service, the text-to-speech service, etc. The punctuation information is obtained by punctuating the first text information, so that the second text information carries the punctuation information.

[0062] Step S140: determining a first waiting time for replying to the user in the dialogue process according to the punctuation information and the request service delay time;

[0063] Since the punctuation information can feedback the speaking state of the user in the dialogue process, it can reduce the situation of being interrupted or interrupting the user, in addition, further considering the request service delay time, comprehensively determining the first waiting time of the customer service robot to reply to the user, it can be understood that when receiving the voice information of the user, the customer service robot needs to wait for the first waiting time to reply to the user, so that the pause time in the voice dialogue becomes effective and controllable, realizing the smooth communication of the user and the customer service robot in the voice dialogue, which is beneficial to improve the user satisfaction and improve the service quality.

[0064] Step S150: matching the second text information with the preset intent library to obtain the reply information of the current round;

[0065] By matching the second text information with the intent library, the reply dialogue corresponding to the second text information can be output, that is, the reply information of the current round is obtained, and the user is replied according to the reply information.

[0066] Step S160: comparing the current round number and the preset reply round number to obtain a comparison result, and replying to the user according to the comparison result, the reply information and the first waiting time, or canceling the reply to the user according to the comparison result and the reply information.

[0067] By further considering the current round number, comparing the current round number and the preset reply round number, obtaining a comparison result, and determining whether to reply to the user or cancel the reply to the user according to the comparison result, the voice dialogue interaction is more intelligent, and the situation of error recognition leading to error reply in the dialogue process is avoided.

[0068] The voice processing method of the above-mentioned first aspect embodiment, by converting the first audio information input by the user into a plurality of first text information and then analyzing, the second text information is obtained by punctuating the first text information, which can lay the foundation for subsequent understanding of the user's voice, based on the full-duplex communication protocol, the second text information is transmitted in multiple rounds, which can improve the speed of responding to the user in the dialogue process, since the second text information carries the key information and punctuation information of the current round, by considering the punctuation information and the request service delay time, the first waiting time of the user in the dialogue process is comprehensively determined, the timing of the customer service robot speaking is realized, which can effectively improve the user's dialogue experience, and further improve the user satisfaction, in addition, by matching the second text information with the intent library, the reply information of the current round can be quickly obtained, by further comparing the current round number and the preset reply round number, the user is determined to wait for the first time to reply or cancel the reply according to the comparison result, so that the user's question in the dialogue process can be more intelligently and efficiently replied.

[0069] As Figure 4As shown, in the above-described speech processing method, step S110 converts the first audio information into multiple first text information, including but not limited to steps S210 and S220:

[0070] Step S210: Convert the first audio information into binary data;

[0071] Step S220: Extract speech features from binary data and input them into a preset ASR speech recognition model to obtain multiple first text information.

[0072] Upon receiving the first audio information from the user, a request is made to the Automatic Speech Recognition (ASR) service to convert the user's audio into text. Specifically, the collected first audio information is processed into binary data, speech features are further extracted, and input into the ASR speech recognition model. Multiple first text information is obtained based on machine learning algorithms and related matching strategies.

[0073] like Figure 5 As shown, in the above-described speech processing method, step S120 involves performing a punctuation service on the first text information to obtain punctuated second text information, including but not limited to steps S310 and S320:

[0074] Step S310: Encode the first text information using a pre-trained BERT model;

[0075] Step S320: Use the BiLSTM+CRF algorithm to predict the punctuation corresponding to the first text information to obtain the second text information, wherein the second text information is a complete sentence with punctuation.

[0076] It's important to note that Bidirectional Encoder Representations from Transformers (BERT) is a deep, bidirectional, unsupervised language representation model that is pre-trained using only a plain text corpus. The punctuation service first encodes the first text information using the BERT pre-trained model, then uses the BiLSTM+CRF algorithm to predict the punctuation corresponding to the first text information, resulting in a complete user question with punctuation, i.e., the second text information. For example, when the user's question "How do I get to place A?" (the first text information) is identified as an interrogative sentence, a question mark is added to the end of the sentence to obtain the second text information, which lays the groundwork for better understanding of the user's speech.

[0077] like Figure 6As shown, in the above-described voice processing method, step S140 determines the first waiting time for responding to the user in the dialogue flow based on punctuation information and the service request delay time, including but not limited to steps S410 to S430:

[0078] Step S410: Determine the user's status information in the current dialogue flow based on the punctuation information;

[0079] Step S420: Determine the second waiting time based on the status information;

[0080] Step S430: Calculate the difference between the second waiting time and the request service delay time to obtain the first waiting time.

[0081] Upon recognizing the second text message in the current round, the system first determines the user's status in the current dialogue flow based on punctuation. This status information can reflect whether the entire dialogue has been completed, or whether the user has only spoken halfway and is still considering their next words. The second waiting time is determined based on this status information. For example, if the second text message is "How do I get to place A?", it indicates a complete statement, allowing for a shorter response time (i.e., a shorter second waiting time). If the second text message is "Let me think…", it indicates the user has more to say, allowing for a longer response time (i.e., a longer second waiting time), giving the user ample time to think and making the dialogue more human-centered. Furthermore, considering the delays caused by requesting other services during the dialogue flow, the difference between the second waiting time and the service request delay is calculated to obtain the first waiting time. This allows the customer service robot to make a comprehensive decision regarding the appropriate response waiting time. For example, if the second waiting time is x seconds and the service request delay time is y seconds, then a reply needs to be given after xy seconds (the first waiting time). That is, a reply is given immediately when xy < 0, which enables the customer service robot to not interrupt, be patient, and listen, greatly improving the user's conversation experience.

[0082] like Figure 7 As shown, in the above-described speech processing method, step S150 matches the second text information with a preset intent library to obtain the response information for the current round, including but not limited to steps S510 and S520:

[0083] Step S510: Perform speech recognition on the second text information to obtain semantic recognition results, wherein the semantic recognition results include user intent information and entity information;

[0084] Step S520: Match the intent information and entity information with the intent database to obtain the response information for the current round.

[0085] When the second text message of the current round is identified, the semantic recognition service identifies the user's intent and possible entity information, thus obtaining the semantic recognition result. By matching the intent and entity information against an intent database, the response information for the current round can be obtained. For example, if the second text message is "How do I get to place A?", the user's intent to "ask for directions" is identified, and the destination (entity information) is "place A". Searching the intent database yields a matching answer. It's worth noting that rule-based matching with models can improve data acquisition speed. By obtaining response information based on intent, entity information, and the intent database, the semantics of the user's voice content can be effectively identified, providing targeted answers and making voice interaction more intelligent.

[0086] like Figure 8 As shown, in the above-described voice processing method, step S160 involves replying to the user based on the comparison result, the reply information, and the first waiting time, or canceling the reply to the user based on the comparison result and the reply information, including but not limited to steps S610 and S620:

[0087] Step S610: If the comparison result shows that the current round number is greater than the preset reply round number, the reply information of the current round is determined to be invalid, and the reply to the user is canceled;

[0088] Step S620: If the comparison result is that the current round number is less than or equal to the preset reply round number, the reply information is converted into second audio information, and after a first waiting time, the second audio information is output to the user.

[0089] If a response is received in the current round, the current round number is compared with the preset response round number. The preset response round number is used to mark the number of rounds (turns) that the voice processing system can respond to. If the current round number is greater than the preset response round number, the response in the current round is considered invalid and will not be replied to. If the current round number is less than or equal to the preset response round number, the response in the current round is considered valid. After a first waiting time, the second audio message is output to the user, making the customer service robot's responses in the dialogue process more effective.

[0090] It should be noted that since the reply information is text information, it is converted into second audio information by requesting the Text To Speech (TTS) service. Specifically, the reply information is first converted into a phoneme sequence and preprocessed. Then, the appropriate basic unit is selected from the massive amount of pre-recorded voice data and concatenated into audio using a splicing method. Finally, it is transmitted to the client 400 and output to the user.

[0091] It should be noted that the service request latency includes the latency incurred when requesting services such as ASR, TTS, and tagging.

[0092] In the voice processing method described above, after obtaining the reply information of the current round in step S150, if third audio information from the user is received, it is determined that the reply information of the current round is invalid, and the user is cancelled.

[0093] It can be understood that, after obtaining the reply information of the current round, it is determined whether third audio information from the user is received in a certain time period, it can be determined whether the user is currently speaking, if the third audio information is received (the client 400 has audio incoming), it is considered that the user is speaking, and it is not appropriate to interrupt the user, then it is considered that the reply information of the current round is invalid, and the user is cancelled, which can realize the function of supporting the user to interrupt in the speaking process of the customer service robot, can give the user the right to speak, greatly improves the dialogue experience of the user and the customer service robot, has great application value in the customer service industry, and plays an extremely important role in improving user satisfaction and service quality of enterprises.

[0094] Based on the voice processing method described above, the following respectively proposes various embodiments of the voice processing system, computer device and computer readable storage medium of the present application.

[0095] As shown in Figure 2 The second aspect embodiment of the present application provides a voice processing system 200 based on a full-duplex communication protocol, which includes but is not limited to an acquisition module 210, a punctuation module 220, a multi-round transmission module 230, a time determination module 240, an information matching module 250, and a comparison module 260. The voice processing system 200 uses a full-duplex communication protocol for data transmission.

[0096] Specifically, the acquisition module 210 is configured to receive first audio information of a user and convert the first audio information into a plurality of first text information; the punctuation module 220 is configured to perform punctuation service on the first text information to obtain second text information with punctuation; the multi-round transmission module 230 is configured to transmit a plurality of the second text information in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time; the time determination module 240 is configured to determine a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time; the information matching module 250 is configured to match the second text information with a preset intent library to obtain reply information of the current round; and the comparison module 260 is configured to compare the current round number with a preset reply round number to obtain a comparison result, and reply to the user according to the comparison result, the reply information and the first waiting time, or cancel the reply to the user according to the comparison result and the reply information.

[0097] The voice processing system 200 of the second aspect embodiment can convert the first audio information input by the user into a plurality of first text information, analyze the first text information, and perform a punctuation service on the first text information to obtain second text information, which can lay a foundation for subsequent understanding of the user's voice. Based on the full-duplex communication protocol, the second text information is transmitted in multiple rounds, which can improve the speed of responding to the user in the dialogue process. Since the second text information carries key information and punctuation information of the current round, the first waiting time for replying to the user in the dialogue process is determined by considering the punctuation information and the request service delay time, and the timing of the customer service robot speaking is calculated individually, which can effectively improve the dialogue experience of the user and further improve the user satisfaction. In addition, by matching the second text information with the intent library, the reply information of the current round can be quickly obtained, and by further comparing the current round number with the preset reply round number, it is determined whether to reply to the user or cancel the reply to the user according to the comparison result, so that the user's question in the dialogue process can be replied more intelligently and efficiently.

[0098] It should be noted that the specific implementation and corresponding technical effects of the voice processing system 200 of the embodiments of the present application can be referred to the specific implementation and corresponding technical effects of the voice processing method described above.

[0099] As shown in Figure 9 The computer device 900 provided by the third aspect embodiment of the present application includes a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920.

[0100] The processor 920 and the memory 910 can be connected by a bus or other means. The memory 910, as a kind of non-transient computer readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 910 can include a high-speed random access memory, and can also include a non-transient memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory 910 can optionally include a memory remotely arranged relative to the processor 920, which can be connected to the processor assembly through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. Those skilled in the art can understand that, Figure 9 The computer device 900 shown in the figure does not constitute a limitation on the embodiments of the present application, and can include more or fewer components than the figure, or combine certain components, or different component arrangements. The non-transient software programs and instructions required to implement the voice processing method of the above embodiments are stored in the memory 910, and when executed by the processor 920, the voice processing method of the above embodiments is executed, for example, the above-describedFigure 1 and Figures 4 to 8 in the method steps.

[0101] The computer device of the third aspect embodiment can convert the first audio information input by the user into a plurality of first text information, analyze the first text information, and perform a punctuation service on the first text information to obtain second text information, which can lay a foundation for subsequent understanding of the voice of the user. The second text information is transmitted in multiple rounds based on the full-duplex communication protocol, which can improve the speed of responding to the user in the dialogue process. Since the second text information carries key information and punctuation information of the current round, the first waiting time for replying to the user in the dialogue process is determined by considering the punctuation information and the request service delay time, which realizes personalized calculation of the speaking time of the customer service robot and effectively improves the dialogue experience of the user and the user satisfaction. In addition, the second text information is matched with the intent library, the reply information of the current round can be quickly obtained, and the user is replied to or the reply to the user is cancelled according to the comparison result of the current round number and the preset reply round number, so that the user's question in the dialogue process can be replied to more intelligently and efficiently.

[0102] In addition, the fourth aspect embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions for executing the voice processing method described above. For example, the voice processing system 200 described above is executed by a processor, so that the processor described above can execute the voice processing method in the embodiments described above, for example, execute the method steps in the above Figure 1 and Figures 4 to 8 The computer device of the third aspect embodiment can convert the first audio information input by the user into a plurality of first text information, analyze the first text information, and perform a punctuation service on the first text information to obtain second text information, which can lay a foundation for subsequent understanding of the voice of the user. The second text information is transmitted in multiple rounds based on the full-duplex communication protocol, which can improve the speed of responding to the user in the dialogue process. Since the second text information carries key information and punctuation information of the current round, the first waiting time for replying to the user in the dialogue process is determined by considering the punctuation information and the request service delay time, which realizes personalized calculation of the speaking time of the customer service robot and effectively improves the dialogue experience of the user and the user satisfaction. In addition, the second text information is matched with the intent library, the reply information of the current round can be quickly obtained, and the user is replied to or the reply to the user is cancelled according to the comparison result of the current round number and the preset reply round number, so that the user's question in the dialogue process can be replied to more intelligently and efficiently.

[0103] As will be appreciated by one of ordinary skill in the art, all or some of the steps, systems, etc. in the above-disclosed methods can be embodied in software, firmware, hardware, and / or suitable combinations thereof. Some or all of the physical components can be implemented with software executed by a processor, such as a central processing unit, a digital signal processor, or microprocessor, or can be implemented with hardware, or can be implemented with an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media), and communication media (or transitory media). As is well known to those of ordinary skill in the art, computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.

[0104] The above description is that of the preferred embodiments of the application. Various modifications and changes can be made thereto without departing from the spirit of the application, which is defined by the appended claims. Each of the additional sequences disclosed herein is individually incorporated by reference in its entirety. Each element of the claims is to be understood, in the alternative, as "means for" performing a function or obtaining a result.

Claims

1. A voice processing method based on a full duplex communication protocol, characterized by, Applied to a voice processing system that adopts a full-duplex communication protocol for data transmission, the voice processing system comprises: receiving first audio information of a user, converting the first audio information into a plurality of first text information; punctuation service is performed on the first text information to obtain second text information with punctuation; based on the full-duplex communication protocol, the plurality of second text information is transmitted in multiple rounds, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time; determining a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time; matching the second text information with a preset intent library to obtain reply information of the current round; comparing the current round number with a preset reply round number to obtain a comparison result, and replying to the user according to the comparison result, the reply information and the first waiting time, or canceling the reply to the user according to the comparison result and the reply information; determining a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time, comprising: determining state information of the user in the current dialogue process according to the punctuation information; wherein the state information is used to reflect whether the second text information is a complete sentence; determining a second waiting time according to the state information; calculating the difference between the second waiting time and the request service delay time to obtain the first waiting time.

2. The voice processing method of claim 1, wherein, converting the first audio information into a plurality of first text information, comprising: converting the first audio information into binary data; extracting speech features from the binary data and inputting them into a preset ASR speech recognition model to obtain a plurality of first text information.

3. The voice processing method of claim 1, wherein, punctuation service is performed on the first text information to obtain second text information with punctuation, comprising: encoding the first text information through a preset BERT pre-training model; predicting punctuation corresponding to the first text information using a BiLSTM+CRF algorithm to obtain second text information, wherein the second text information is a complete sentence with punctuation.

4. The voice processing method of claim 1, wherein, matching the second text information with a preset intent library to obtain reply information of the current round, comprising: performing speech recognition on the second text information to obtain semantic recognition results, wherein the semantic recognition results include user intent information and entity information; matching the intent information and the entity information with the intent library to obtain reply information of the current round.

5. The voice processing method of claim 1, wherein, replying to the user according to the comparison result, the reply information and the first waiting time, or canceling the reply to the user according to the comparison result and the reply information, comprising: if the comparison result is that the current round number is greater than the preset reply round number, it is determined that the reply information of the current round is invalid, and the reply to the user is canceled; if the comparison result is that the current round number is less than or equal to the preset reply round number, the reply information is converted into second audio information, and the second audio information is output to the user after pausing for the first waiting time.

6. The voice processing method of claim 1, wherein, After the reply information of the current round is obtained, if third audio information from the user is received, it is determined that the reply information of the current round is invalid, and the user is cancelled.

7. A voice processing system based on a full duplex communication protocol, characterized by The voice processing system adopts a full-duplex communication protocol for data transmission, including: An acquisition module is configured to receive first audio information of a user and convert the first audio information into a plurality of first text information. A punctuation marking module is configured to perform punctuation marking service on the first text information to obtain second text information with punctuation. A multi-round transmission module is configured to transmit a plurality of the second text information in multiple rounds based on the full-duplex communication protocol, wherein the second text information carries key information and punctuation information of the current round, and the key information includes a current round number and a request service delay time. A time determination module is configured to determine a first waiting time for replying to the user in a dialogue process according to the punctuation information and the request service delay time. An information matching module is configured to match the second text information with a preset intent library to obtain reply information of the current round. A comparison module is configured to compare the current round number with a preset reply round number to obtain a comparison result, and reply to the user according to the comparison result, the reply information and the first waiting time, or cancel the reply to the user according to the comparison result and the reply information. The voice processing system is further configured to: Determine state information of the user in the current dialogue process according to the punctuation information, wherein the state information is used to reflect whether the second text information is a complete sentence; determine a second waiting time according to the state information; calculate a difference between the second waiting time and the request service delay time to obtain the first waiting time.

8. A computer device, comprising: Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the voice processing method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer executable instruction is stored, and the computer executable instruction is used to execute the voice processing method of any one of claims 1 to 6.

Images