Quietness detection methods, storage media and electronic devices

Abstract

This application provides a silence detection method, storage medium, and electronic device. The method acquires call audio, performs structured text transcription on the audio to obtain structured text, and detects whether the customer service representative's speech contains an intent to waive restrictions requiring additional processing time. If such an intent exists, a preset exemption period is activated starting from the end time of the customer service representative's speech. For the customer's speech, if the start time is within the exemption period, the silence timer for the customer's speech is skipped; if the start time is outside the exemption period, the silence timer for the customer's speech is started. If the silence time obtained from the silence timer exceeds a preset silence threshold, or if the customer service representative has not responded by the end of the exemption period, a silence event is determined to exist. This application's embodiment, through an exemption intent recognition mechanism, flexibly adjusts the silence detection threshold, effectively filters reasonable silence, and significantly improves the reliability and business adaptability of the detection results.

Description

Technical Field

[0001] This application relates to the field of audio signal processing technology, and more specifically, to a silence detection method, a storage medium, and an electronic device. Background Technology

[0002] Timely response from hotline customer service is one of the core performance indicators, and abnormal silence / mutes during customer service calls directly impact customer experience. Currently, mainstream silence detection technologies fall into two main categories, but both have significant limitations:

[0003] The first category is indirect analysis methods based on ASR-transcribed text. This method first converts audio into text using Automatic Speech Recognition (ASR), and then analyzes the silence interval based on the timestamps of the text. However, ASR is prone to missing short responses in dialogues, and when users interrupt, the transcribed text may contain logical errors in the dialogue sequence, leading to false silence detections. In addition, current text-based silence detection lacks fault tolerance mechanisms, fails to understand business scenarios, and lacks the ability to distinguish necessary and reasonable waiting in customer service work. A large number of "reasonable silences" are misjudged as "service violations," resulting in unreliable silence detection results.

[0004] The second category is detection methods based on audio signal analysis. These methods only identify silent segments in the audio stream, and their endpoint detection accuracy is insufficient, making it impossible to detect brief responses during dialogue, ultimately leading to inaccurate silence detection.

[0005] Improving the accuracy of noise detection is an urgent problem that needs to be solved. Summary of the Invention

[0006] The main objective of this application is to provide a method for detecting silence, a storage medium, and an electronic device, so as to at least solve the problem of insufficient accuracy in silence detection in the prior art.

[0007] To achieve the above objectives, according to one aspect of this application, a method for detecting silence is provided, comprising:

[0008] The call audio is acquired, and the call audio is processed by structured text transcription to obtain structured text, which includes customer service statements, customer statements, and timestamps corresponding to each statement.

[0009] The system detects whether there is an intention to waive the requirement for additional processing time in the customer service statement. If such an intention exists, a waiver period of a preset duration is activated starting from the end time of the customer service statement.

[0010] For the customer's speech, if the start time of the customer's speech is within the exemption period, the mute timer for the customer's speech is skipped; if the start time of the customer's speech is outside the exemption period, the mute timer for the customer's speech is applied.

[0011] If the silence time obtained by the silence timer is greater than the preset silence threshold, or if customer service has not responded by the end of the exemption period, a silence event is confirmed.

[0012] Optionally, detecting whether the customer service statement contains an intention to waive the requirement for additional processing time includes:

[0013] The customer service statement is subjected to regular expression matching. If a preset exemption statement is matched, it is determined that the customer service statement contains the exemption intention.

[0014] Optionally, detecting whether the customer service statement contains an intention to waive the requirement for additional processing time includes:

[0015] Semantic recognition is performed on the customer service statement. If a preset exemption semantic is identified, it is determined that the customer service statement contains the exemption intention.

[0016] Optionally, the mute timing for the customer's speech includes:

[0017] Calculate the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer's statement after the customer's speech, and use the time difference as the silence time.

[0018] Optionally, the exemption period has an exemption duration, which is determined based on the complexity of the customer's problem and / or the customer service representative's attributes;

[0019] The complexity of customer issues includes: simple issues, routine issues, and complex issues;

[0020] The customer service attributes include: customer service level and customer service proficiency in the business area.

[0021] Optionally, the method further includes:

[0022] If a new customer service message is detected during the exemption period, the exemption period will be terminated.

[0023] Optionally, the method further includes:

[0024] If the new customer service message contains the intention to waive the waiver, a new waiver period will be reactivated starting from the end time of the new customer service message.

[0025] Optionally, the method further includes:

[0026] If the silence time obtained by the silence timer is greater than the silence threshold, a silence mark is added to the customer's speech;

[0027] If the customer service representative does not respond by the end of the exemption period, the customer service representative's message corresponding to the exemption period will be marked with the mute mark.

[0028] Optionally, the method further includes:

[0029] The customer's speech with the mute tag is taken as the target customer's speech. The dialogue flow logic is checked on the target customer's speech to detect whether the previous sentence of the target customer's speech was a customer service speech.

[0030] If the previous sentence of the target customer's statement was a customer service statement, and the end time of the target customer's statement is earlier than the end time of the previous customer service statement, then the mute mark of the target customer's statement is removed.

[0031] Optionally, the method further includes:

[0032] After performing the dialogue flow logic verification, for the target customer who still has the silence mark speaking, detect whether there are preset customer service voice features in the context audio segment corresponding to the target customer's speech in the customer service audio;

[0033] If the customer service voice feature is detected in the context audio segment, the mute marker of the target customer's speech is removed, the customer service audio is re-transcribed into structured text, and the mute detection method is executed again on the re-transcribed structured text.

[0034] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the described silent detection methods.

[0035] According to another aspect of this application, an electronic device is provided, comprising:

[0036] One or more processors, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs being used to perform any of the described silence detection methods.

[0037] By applying the technical solution of this application, call audio is acquired, and structured text transcription is performed on the call audio to obtain structured text. The structured text includes customer service statements, customer statements, and timestamps corresponding to each statement. The system detects whether the customer service statements contain an intention to waive additional processing time. If such an intention exists, a waiver period of a preset duration is activated, starting from the end time of the customer service statement. For customer statements, if the start time of the customer statement is within the waiver period, the silence timer for the customer statement is skipped; if the start time of the customer statement is outside the waiver period, a silence timer is applied to the customer statement. If the silence time obtained from the silence timer exceeds a preset silence threshold, or if the customer service representative has not responded by the end of the waiver period, a silence event is confirmed. This application embodiment, through a waiver intention recognition mechanism, flexibly adjusts the silence detection threshold, effectively filters reasonable silence, and significantly improves the reliability and business adaptability of the detection results.

[0038] This application also introduces a dialogue flow logic verification mechanism, which detects and corrects word order anomalies in structured text by comparing precise timestamps and eliminates false silence markers caused by interruptions, thereby significantly improving detection accuracy.

[0039] This application also introduces a short response recognition mechanism to accurately identify short human voice responses on the customer service side, thereby achieving effective detection and correction of short responses. Attached Figure Description

[0040] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 A hardware structure block diagram of a mobile terminal performing a silence detection method according to an embodiment of this application is shown;

[0042] Figure 2 A schematic flowchart of a noise detection method according to an embodiment of this application is shown;

[0043] Figure 3 A flowchart illustrating the exemption state management mechanism in a silence detection method according to an embodiment of this application is shown.

[0044] Figure 4 A schematic flowchart of dialogue flow logic verification in a silence detection method according to an embodiment of this application is shown;

[0045] Figure 5 A schematic flowchart illustrating the output of detection results in a noise detection method according to an embodiment of this application is shown.

[0046] The above figures include the following reference numerals:

[0047] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0051] As described in the background section, existing silence detection schemes suffer from problems such as incorrect identification of user interruption timing, inability to distinguish between necessary waiting time and service errors, and inability to detect brief customer service responses, ultimately leading to inaccurate silence detection. To address the problem of insufficient accuracy in silence detection, embodiments of this application provide a silence detection method, a storage medium, and an electronic device.

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0053] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a silent detection method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0054] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the silent detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0055] This embodiment provides a method for detecting silence on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] Figure 2 This is a flowchart of a noise detection method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0057] Step S201: Obtain the call audio and perform structured text transcription on the call audio to obtain structured text.

[0058] Specifically, the aforementioned call audio includes voice data of conversations between customer service representatives and customers. Before transcribing the call audio into structured text, preprocessing can be performed on the call audio, such as filtering out unnecessary content for transcription, like music and noise.

[0059] In practice, Automatic Speech Recognition (ASR) can be used to process the call audio into structured text, resulting in structured text. It should be noted that the specific method of structured text transcription can be chosen according to requirements; this embodiment does not impose any specific limitations on this.

[0060] After transcribing the call audio into structured text, the structured text contains the customer service representative's statements, the customer's statements, and the timestamps corresponding to each statement. Below is an example of structured text, where `content` represents the dialogue content; `dialogIdIndex` represents the sentence ID; `startTime` is the start time of the dialogue content; `endTime` is the end time of the dialogue content; and `sender` is the role of the speaker (e.g., customer service representative or customer).

[0061] {

[0062] "content": "Hello, this is Gree Customer Service Center. How can I help you?"

[0063] "dialogIdIndex": "1",

[0064] "startTime": "2025-10-11 00:17:35",

[0065] "endTime": "2025-10-11 00:17:37",

[0066] "sender": "Customer Service",

[0067] },

[0068] {

[0069] "content": "Hello, do you offer free air conditioner cleaning now?"

[0070] "dialogIdIndex": "2",

[0071] "startTime": "2025-10-11 00:17:38",

[0072] "endTime": "2025-10-11 00:17:41",

[0073] "sender": "customer",

[0074] },

[0075] {

[0076] "content": "My air conditioner needs cleaning".

[0077] "dialogIdIndex": "3",

[0078] "startTime": "2025-10-11 00:17:41",

[0079] "endTime": "2025-10-11 00:17:43",

[0080] "sender": "customer",

[0081] },

[0082] {

[0083] "content": "Yes, is it a floor-standing or wall-mounted air conditioner?"

[0084] "dialogIdIndex": "4",

[0085] "startTime": "2025-10-11 00:17:49",

[0086] "endTime": "2025-10-11 00:17:50",

[0087] "sender": "Customer Service",

[0088] },

[0089] {

[0090] "content": "cabinet air conditioner",

[0091] "dialogIdIndex": "5",

[0092] "startTime": "2025-10-11 00:17:51",

[0093] "endTime": "2025-10-11 00:17:52",

[0094] "sender": "customer",

[0095] },

[0096] {

[0097] "content": "Okay, what is your address? We'll register you here."

[0098] "dialogIdIndex": "6",

[0099] "startTime": "2025-10-11 00:18:10",

[0100] "endTime": "2025-10-11 00:18:15",

[0101] "sender": "customer",

[0102] }

[0103] Step S202: Detect whether there is an intention to waive the requirement for additional processing time in the customer service statement. If there is an intention to waive the requirement, activate the waiver period of the preset duration starting from the end time of the customer service statement.

[0104] Exemption from intent detection is designed to address the issue of the basic silence detection rules being too simplistic and inflexible in customer service silence detection, leading to a large number of instances of customer service staff muting. This can improve the rationality of silence detection.

[0105] In actual customer service interactions, after a user speaks, customer service representatives may need to query information or work orders. It's necessary to allow them time for these queries to improve the system's operational rationality and fairness. Therefore, in step S202, exemption intent is detected in the customer service representative's statement. If exemption intent is detected, the exemption duration is determined.

[0106] Specifically, exemption intent detection methods such as regular expression matching, semantic similarity detection, or a combination of regular expression matching and semantic similarity matching can be used to identify exemption intent in customer service statements. In practice, regular expression matching should be prioritized due to its speed; once a match is found, the result is returned immediately. If no regular expression match is found, semantic matching is executed to ensure that non-standard expressions are also recognized.

[0107] It should be noted that the above rules for identifying exemption intent can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on this.

[0108] When a customer service representative's message is detected to have an intent to grant an exemption, an exemption period is activated. This exemption period has a duration TD, meaning it begins at the end time of the message and ends after TD. The exemption period applies only to the customer service representative; during this period, if a new customer message appears, the representative is allowed not to respond to it.

[0109] In practice, the exemption duration can be determined based on a pre-set exemption duration setting strategy.

[0110] It should be noted that the above-mentioned exemption duration setting strategy can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on it.

[0111] Step S203: For customer speech, if the start time of the customer speech is within the exemption period, skip the mute timer for the customer speech; if the start time of the customer speech is outside the exemption period, start the mute timer for the customer speech.

[0112] In step S202, the triggering mechanism for the exemption period has been determined. For customer service statements with an intention to grant an exemption, an exemption period is activated for each customer service statement.

[0113] Normally, when a customer speaks, the mute timer starts from the end time of their message. However, in the exemption mechanism described in this embodiment, if the customer's start time falls within a certain exemption period, customer service is allowed not to reply to the customer's message, i.e., the mute timer for that customer's message is skipped. After the exemption period ends, the mute timer starts again for new customer messages that occur after the end time of the exemption period.

[0114] Step S204: If the silence time obtained by the silence timer is greater than the preset silence threshold, or if customer service has not responded by the end of the exemption period, it is determined that a silence event has occurred.

[0115] It should be noted that the above-mentioned mute threshold can be set according to needs, such as 5 seconds, 10 seconds, etc., and this embodiment does not impose specific restrictions on it.

[0116] Through the above embodiments, call audio is acquired, and structured text transcription is performed on the call audio to obtain structured text. The structured text includes customer service statements, customer statements, and timestamps corresponding to each statement. The system detects whether the customer service statements contain an exemption intent indicating the need for additional processing time. If an exemption intent exists, a preset exemption period is activated starting from the end time of the customer service statement. For customer statements, if the start time of the customer statement is within the exemption period, the silence timer for the customer statement is skipped; if the start time of the customer statement is outside the exemption period, the silence timer for the customer statement is started. If the silence time obtained from the silence timer exceeds a preset silence threshold, or if the customer service representative has not responded by the end of the exemption period, a silence event is determined to exist. This application embodiment, through an exemption intent recognition mechanism, flexibly adjusts the silence detection threshold, effectively filters reasonable silence, and significantly improves the reliability and business adaptability of the detection results.

[0117] In one implementation, detecting whether a customer service representative's statement contains an intention to grant an exemption that requires additional processing time includes: performing regular expression matching on the customer service statement; if a preset exemption statement is matched, it is determined that the customer service representative's statement contains an intention to grant an exemption. This embodiment uses a regular expression matching strategy, which is fast and returns the result immediately upon successful matching, thereby improving the overall system response speed.

[0118] Specifically, for regular expression matching, a pre-defined regular expression can be used to quickly match customer service statements. This embodiment defines the following four exemption modes. If a match is successful, an exemption period is activated, and the specific exemption mode and the matched text fragment of the customer service statement are recorded. The exemption modes include:

[0119] (1) Query mode: Matches customer service representatives who clearly express their intention to query, verify, or confirm, such as "I'll check" or "I'll verify and see".

[0120] (2) Time estimation mode: Match customer service to inform customers of the waiting time, such as "wait a minute" or "it will take a little time".

[0121] (3) System status pattern: Match customer service to explain the waiting expression due to technical reasons such as system and network, such as "the system is loading" or "the network is a bit slow".

[0122] (4) Process description mode: Match the customer service description of the process being processed, such as "processing for you" or "I will help you operate it right away".

[0123] In one implementation, detecting whether there is an exemption intention in the customer service statement that indicates the need for additional processing time includes: performing semantic recognition on the customer service statement, and if a preset exemption semantic is identified, determining that there is an exemption intention in the customer service statement.

[0124] Specifically, this embodiment primarily employs a semantic similarity matching strategy. This strategy involves constructing an exemption intent semantic core library, encompassing various semantic cores related to queries, waiting, and processing. Next, text vectorization is performed, using a pre-trained language model to convert customer service statements and exemption intent core expressions into high-dimensional semantic vectors. Then, the cosine similarity between the customer service statement vector and each exemption intent core vector is calculated, and the highest similarity value is taken as the matching score. For example, a similarity threshold of 0.75 can be set; when the matching score exceeds this threshold, the customer service statement is deemed to contain an exemption intent.

[0125] In practice, regular expression matching can be used to quickly identify statements with obvious exemption intent. Then, a semantic similarity matching strategy can be used to further filter the remaining customer service statements whose meanings meet the exemption criteria but whose expression is not covered by the regular expression pattern. This ensures the accuracy of the exemption intent identification results. For example, the following is an example of exemption intent detection:

[0126] Customer service replied: "Please be patient, I'll check the system."

[0127] Regular expressions might not be able to match this colloquial expression. In this case, the system calculates semantic similarity and finds that the vector similarity between the sentence and the core expression of the exemption intent, "Please wait a moment, let me check the system," reaches 0.82, exceeding the threshold of 0.75. Therefore, the system successfully determines that the sentence contains an exemption intent and activates the exemption period.

[0128] In one implementation, timing the silence of a customer's speech includes: calculating the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer statement after the customer's speech, and using the time difference as the silence time.

[0129] The silence time can be expressed by the following formula:

[0130] ,

[0131] in, The end time for the customer's speech. The start time of the first customer service representative's statement after the customer's statement, or the first customer's statement. Indicates the duration of silence.

[0132] like >T, where T represents the preset mute threshold. This means that the customer service representative has not responded to this customer message for a long time. At this time, a mute mark is added to this customer message. The mute mark indicates that the customer service representative is suspected of muteing this customer message once.

[0133] like This indicates that customer service responded promptly or that the customer spoke continuously.

[0134] This embodiment uses the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer's statement after the customer's speech as the silence time, which reasonably describes the time range in which silence events may occur in actual dialogue scenarios, thereby enabling accurate identification of silence events.

[0135] In one implementation, the exemption period has an exemption duration, which is determined based on the complexity of the customer's problem and / or the customer service representative's attributes; wherein, the complexity of the customer's problem includes: simple problems, routine problems, and complex problems; and the customer service representative's attributes include: customer service representative level and customer service representative's proficiency in the business area.

[0136] In practice, the exemption period can be automatically adjusted based on the complexity of the customer's question and / or the customer service representative's attributes. Taking home appliance customer service as an example, common customer question types include pre-sales product inquiries, price and model comparisons, after-sales installation appointment inquiries, fault reporting, and complaint feedback. If the customer service representative is a senior pre-sales representative with many years of experience, the exemption period can be reduced according to the preset adjustment strategy when a customer inquires about pre-sales issues; if the customer service representative is a new after-sales representative, the exemption period should be extended when a customer inquires about pre-sales issues. If the customer service representative is only making a simple inquiry, the exemption period should be reduced; if complex operations are required, a longer exemption period should be granted.

[0137] To adapt to the varying management granularity requirements of different business scenarios, this embodiment provides multiple configurable exemption duration setting strategies. These are illustrated below through three specific scenario examples.

[0138] Example scenario 1: Setting an exemption duration strategy based on the complexity of the customer's problem.

[0139] This exemption duration setting strategy is suitable for business scenarios that prioritize simple, stable, and easily manageable rules. This strategy sets a fixed exemption duration based solely on the estimated complexity of the customer's problem, without considering individual customer service differences. Please refer to Table 1, which schematically illustrates one exemption duration setting strategy.

[0140] Table 1

[0141]

[0142] Table 1 categorizes issues into simple queries, routine processing, and complex operations based on their complexity. Table 1 also shows typical scenario descriptions, exemption durations, and specific operations that customer service representatives need to perform for different issue types.

[0143] Taking the exemption time setting strategy shown in Table 1 as an example, if a customer asks, "I lost the remote control for my air conditioner, how much does it cost to replace it?", the system recognizes this as a simple query and automatically sets a 30-second exemption time. Customer service needs to check the price of the replacement part and reply within this time. The advantage of this exemption time setting strategy is that the rules are transparent and the computational overhead is minimal.

[0144] Scenario Example 2: Fixed time settings based on customer service attributes.

[0145] This exemption duration setting strategy is suitable for scenarios that emphasize personalized management and fairness, have a highly homogeneous business type, and exhibit minimal differences in issue complexity. The exemption duration is primarily set based on the customer service personnel's own attributes. Its advantage lies in its extremely simple rules, but it does not consider differences in issue complexity. Please refer to Table 2, which schematically illustrates one exemption duration setting strategy.

[0146] Table 2

[0147]

[0148] Table 2 shows that the adjustment factors for the exemption duration are related to the customer service level and the matching degree of the customer service's business area. When calculating the exemption duration, both the customer service level and the matching degree of the customer service's business area need to be considered. Table 2 shows the specific exemption duration for different customer service levels and different business area matching degrees, as well as the relevant explanations for setting the exemption duration.

[0149] The calculation formula for the exemption duration is as follows: Exemption duration = Exemption duration corresponding to customer service level + Adjustment time corresponding to business area matching.

[0150] Taking a junior customer service representative handling a cross-domain issue as an example, the exemption time for junior customer service representatives is 40 seconds, and the exemption time for business domain matching is 15 seconds. Therefore, the final exemption time is: 40 seconds + 15 seconds = 55 seconds.

[0151] Scenario Example 3: Weighted settings for comprehensive question types and customer service attributes.

[0152] This embodiment, based on distinguishing problem types, further considers the differences in customer service personnel capabilities to set a baseline exemption duration and adjustment coefficient, making the exemption duration more reasonable while adapting to business needs. Please refer to Table 3, which schematically illustrates an exemption duration setting strategy for this scenario.

[0153] Table 3

[0154]

[0155] Table 3 shows that the adjustment factors for the exemption duration are related to the problem type, customer service level, and the degree of matching between the customer service representative's business area and the exemption duration. When calculating the exemption duration, it is necessary to consider the problem type, customer service level, and the degree of matching between the customer service representative's business area and the exemption duration simultaneously. Table 3 shows the specific exemption duration for different problem types, different customer service levels, and different degrees of matching between business areas, as well as the relevant explanations for setting the exemption duration.

[0156] The calculation formula for the exemption duration is as follows: Exemption duration = Exemption duration corresponding to the problem type × Customer service level coefficient + Business area matching adjustment time.

[0157] Taking a senior customer service representative handling a complex cross-domain operational issue as an example, the exemption time corresponding to the issue type is 90 seconds, the senior customer service representative's coefficient is 0.9, and the business domain matching adjustment time is increased by 15 seconds, resulting in a final exemption time of 90 seconds. 0.9 + 15 seconds = 96 seconds.

[0158] By flexibly setting the exemption duration, it is possible to fully adapt to the exemption requirements of different business scenarios, thereby improving the accuracy of the overall silence detection method.

[0159] It should be noted that the exemption duration setting strategy in this embodiment can be set according to needs and is not limited to the three exemption duration setting strategies mentioned above.

[0160] In one implementation, an exemption status management mechanism can also be set up. Specifically, if a new customer service message is detected during the exemption period, the exemption period is terminated.

[0161] Specifically, the exemption status management mechanism also includes: if a new customer service message contains an intention to grant an exemption, a new exemption period will be reactivated starting from the end time of the new customer service message.

[0162] For example, such as Figure 3 As shown, the process of the exemption status management mechanism is as follows:

[0163] Step S301: When an exemption intent is detected, the exemption period is activated starting from the end of the current customer service speech.

[0164] Step S302: During the exemption period, no mute timer is set for new customer service messages.

[0165] Step S303: If the customer service representative responds to the customer within the exemption period, the exemption period ends immediately.

[0166] Step S304: If the exemption period expires and customer service still does not respond, the exemption status is terminated and it is immediately determined that a mute event has occurred.

[0167] Step S305: Output the judgment result of the silence detection.

[0168] For example, the following uses "Air Conditioner Repair Progress Inquiry" as an example to illustrate... Figure 3 The exemption status management mechanism is explained in detail. Sentences 3 to 5 in the "Air Conditioner Repair Progress Inquiry" section are as follows:

[0169] Sentence 3: The customer service representative said, "Let me check the progress of your work order; it will take a little time."

[0170] Sentence 4: The customer says, "Okay, also, my air conditioner is making strange noises."

[0171] Sentence 5: Customer service representative's message: "Found it, your work order is being processed. Regarding the abnormal noise issue you just mentioned, I need to understand it in more detail."

[0172] Sentence 3 is a customer service statement, so the first step is to check if sentence 3 contains an intention to grant an exemption. Since the keywords "check it out" and "it will take a little time" were detected, it can be determined that sentence 3 contains an intention to grant an exemption. Taking the above scenario example 3 as an example, combining the problem type (routine processing, 60 seconds) and the customer service level (intermediate, coefficient 1.0), and setting the exemption time to 60 seconds, the exemption period is represented as follows: , This indicates the end time of sentence 3, in seconds.

[0173] Next, regarding sentence 4, assuming that sentence 4 occurs within the exemption period, then silence detection will not be performed on sentence 4 during the exemption period.

[0174] Finally, regarding sentence 5, assuming that the customer service representative replied 45 seconds after the start of the exemption period, then the customer service representative's reply was within the exemption period, and the exemption status terminated.

[0175] This embodiment sets up a complete management mechanism of "activation → termination → reactivation" for the exemption period, which fully adapts to all possible situations during the exemption period, improves the flexibility and rationality of the exemption period mechanism, and thus improves the accuracy of the silence detection results.

[0176] In one embodiment, the above-mentioned silence detection method further includes: if the silence time obtained by the silence timing is greater than the silence threshold, marking the customer's speech with a silence mark; if the customer service representative still has not responded by the end of the exemption period, marking the customer service representative's speech corresponding to the exemption period with a silence mark. This embodiment achieves accurate positioning of different silence events by adding silence marks, which facilitates subsequent review of the silence detection results.

[0177] In one embodiment, the above-mentioned silence detection method further includes: taking a customer's speech with a silence mark as the target customer's speech, performing a dialogue flow logic check on the target customer's speech, and detecting whether the previous sentence of the target customer's speech was a customer service speech; if the previous sentence of the target customer's speech was a customer service speech, and the end time of the target customer's speech is earlier than the end time of the previous customer service speech, canceling the silence mark of the target customer's speech.

[0178] In practice, user interruptions may occur, leading to disordered sentence order in the structured text. This is because in actual conversations, the same speaker may pause naturally, or one party may interrupt the other. An ideal Automatic Speech Recognition (ASR) system should be able to segment sentences containing long pauses appropriately and accurately present interaction sequences such as "customer service - customer (interruption) - customer service". However, ASR models often suffer from the following two types of problems due to coarse or incorrect sentence segmentation granularity:

[0179] 1. Incorrectly merging spoken content from the same person, including natural pauses, into a single text;

[0180] 2. Cover up other people's interruptions within the same speaker's text block.

[0181] Both of these errors distort the original temporal logic of the dialogue, leading to misjudgments in silence detection.

[0182] Therefore, this embodiment proposes a dialogue flow logic verification mechanism. Its core principle lies in using the precise start and end timestamps of each sentence in the structured text to detect timing inconsistencies, thereby inferring potential merging or segmentation errors in ASR transcription. The specific verification rule is as follows: when a silence marker is detected for a customer's statement, if the preceding sentence is a customer service statement, and the end time of this customer's statement is earlier than the end time of the previous customer service statement (customer interruption), then it indicates an ASR transcription anomaly or customer interruption. In this case, the system will rule out the possibility of customer service silence and remove the silence marker from that customer's statement.

[0183] For example, as shown in Table 4, Table 4 presents a structured text of a conversation between a customer service representative and a customer to verify the delivery address.

[0184] Table 4

[0185]

[0186] In response to this dialogue, execute as follows Figure 4 The silence detection process shown, which includes a dialogue flow logic verification mechanism, includes the following steps:

[0187] Step S401, Initial silence detection.

[0188] For customer sentence 5 (content "201") in Table 4, calculate the interval between its end time and the start time of the next sentence (customer sentence 6): ΔT = t_start(sentence 6) - t_end(sentence 5) = 00:17:55 - 00:17:42 = 13 seconds.

[0189] Assuming the silence threshold T = 5 seconds, since 13 seconds > 5 seconds, the system initially flags the customer service representative for a single instance of silencing the customer in sentence 5.

[0190] Step S402: Trigger dialogue flow logic verification.

[0191] The system applies validation rules to check the contextual timing of the client's sentence 5:

[0192] Check if the sentence preceding customer sentence 5 (sentence 4) was spoken by customer service. The result is yes.

[0193] Check if the end time of customer sentence 5 (00:17:42) is earlier than the end time of the previous customer service statement (sentence 4) (00:17:52). Since 00:17:42 < 00:17:52, the result is yes.

[0194] Step S403, Analysis and Judgment.

[0195] The aforementioned temporal inconsistency (the customer finished speaking before the customer service representative finished speaking) reveals that the actual dialogue process should have been as follows: after the customer service representative repeated the address "Building 5," the customer immediately interrupted to confirm "201," and then the customer service representative continued with the subsequent remarks. The ASR system incorrectly merged the two segments of the customer service representative's speech before and after the interruption into sentence 4, thus concealing the fact of the interruption and causing sentence 5 to be misjudged as an independent, unanswered customer question.

[0196] Step S404: Perform the correction.

[0197] Based on the dialogue flow verification results, the system determined that this silence marker was caused by an ASR transcription anomaly (the interruption was not correctly segmented), and not by a customer service error. Therefore, the silence marker for customer sentence 5 was removed, and the silence detection result was corrected.

[0198] Through this mechanism, this embodiment can effectively identify and correct mute misjudgments caused by ASR sentence merging, interruption and segmentation errors, etc., and significantly improve the accuracy and reliability of mute detection in complex dialogue interaction scenarios.

[0199] In one implementation, the above-mentioned silence detection method also includes a short response recognition mechanism. This mechanism addresses silence situations caused by short customer service responses being missed during ASR transcription. Specifically, during actual conversations, while the customer is describing their situation, the customer service representative is often in a state of reassuring or listening to the customer, and frequently provides short responses. These short responses are easily missed during ASR transcription, and thus, during text-based detection, the customer service representative may be mistakenly judged as violating the silence rule.

[0200] To address this issue, a brief response recognition mechanism can be adopted, which includes: after performing dialogue flow logic verification, for the target customer's speech that still has a silence mark, detecting whether there are preset customer service voice features in the context audio segment corresponding to the target customer's speech in the customer service audio; if customer service voice features are detected in the context audio segment, canceling the silence mark of the target customer's speech, re-transing the customer service audio into structured text, and re-executing the silence detection method on the re-transcribed structured text.

[0201] In practice, call audio is handled in two ways. The first is dual-channel, which includes two independent audio streams: customer audio and customer service audio. The customer audio only contains all the sounds from the customer service side. The second is single-channel, which has only one mixed audio stream, where the customer service and customer voices are recorded together. The audio includes all the sounds from both sides.

[0202] For example, for dual-channel call audio, a short response recognition mechanism includes:

[0203] For target customer statements that still retain a silence flag after the dialogue flow logic verification, an audio segment within the first time interval is selected from the customer service audio as the context audio segment corresponding to the target customer statement. The presence of preset human voice features is then detected within this context audio segment. The first time interval is represented as [t1, t2 + T], where t1 represents the start time of the target customer service statement, t2 represents the end time of the target customer service statement, and T represents the silence threshold. If preset human voice features are detected in the context audio segment, the customer service audio is re-transcribed into structured text, and the aforementioned silence detection method is repeated on the re-transcribed structured text; otherwise, the silence flag for the target customer statement is maintained.

[0204] Specifically, the aforementioned voice features refer to customer service voice response features. Customer service voice responses typically contain specific short customer service phrases. Therefore, customer service voice response features can be constructed based on this characteristic to distinguish them from environmental noise features in call audio. For example, the specific short customer service phrase features shown in Table 5 can be used.

[0205] Table 5

[0206]

[0207]

[0208] Based on the customer service voice response features described in Table 5, five feature dimensions of the customer service speech are detected in the first time interval [t1, t2+T], including:

[0209] Duration detection: A short-time energy thresholding method is used. First, the short-time energy of audio frames within a first time interval is calculated. A noise floor threshold is set (e.g., -40dB), and continuous speech segments with frequencies above the noise floor threshold are extracted. The total duration of these continuous speech segments is then calculated. If it falls within the duration range of "customer service short phrase characteristics": 120ms≤ If the duration is ≤700ms, the duration detection result is recorded as "satisfied"; otherwise, it is recorded as "not satisfied".

[0210] Fundamental frequency detection: Fundamental frequency estimation is performed using the autocorrelation function method or the YIN algorithm. If a valid fundamental frequency is detected... It falls within the male (80-180Hz) or female (160-320Hz) range, and the fundamental frequency change rate within the segment is... If the frequency is less than the preset threshold, it indicates a constant frequency of non-mechanical noise, and the result is recorded as "satisfied" for the fundamental frequency detection; otherwise, it is recorded as "not satisfied".

[0211] Energy detection: The energy spectrum is calculated using short-time Fourier transform, and the energy envelope is extracted. The difference between the maximum energy Em within a segment and the background noise E is calculated. .like If the value is greater than 6dB and the variance σE of the energy envelope is greater than the preset threshold, it indicates fluctuations and non-stationary noise. In this case, the energy detection result is recorded as "satisfied"; otherwise, it is recorded as "unsatisfied".

[0212] Zero-crossing rate detection: Directly count the number of zero-crossing points in the audio frame and calculate the average value ZCRavg per unit time. If ZCRavg falls in the voiced area or transition area, or shows obvious segmented changes, such as high at the beginning and low at the end, and there are no constant extreme values ​​across the entire frequency band, such as the constant median of white noise, then the zero-crossing rate detection result is recorded as "satisfied". Otherwise, it is recorded as "not satisfied".

[0213] Formant detection: Linear predictive coding (LPC) analysis is used to extract the frequencies and bandwidths of the first 4-5 formants. If at least 2 significant formant peaks are detected, and their frequency positions conform to the typical distribution of human vowels or nasal sounds, such as F1 between 300-1000Hz, and the formant frequency changes smoothly over time, then the formant detection result is recorded as "satisfied"; otherwise, it is recorded as "not satisfied".

[0214] After the testing is completed, a comprehensive judgment is made, and the test results of the above 5 dimensions are weighted and scored. Either of the following two methods can be used:

[0215] (1) Full match mode: If the detection results of all 5 dimensions are marked as "satisfied", then [t1,t2+T] is directly determined as "suspected customer service response". At this time, the structured text transcription process needs to be re-executed, and the above-mentioned silence detection method is executed again on the re-transcribed structured text.

[0216] (2) Weighted threshold mode: Considering the complexity of the actual environment, a certain degree of fault tolerance is allowed. The system sets a weight for each dimension (e.g., fundamental frequency 0.3, formant 0.25, duration 0.20, energy 0.15, zero crossing rate 0.10). If the number of feature dimensions is ≥4, or the total weighted score is ≥0.8 (out of 1.0), then [t1, t2+T] is determined to be "suspected customer service response". At this time, the structured text transcription process needs to be repeated, and the above-mentioned silence detection method is executed again on the re-transcribed structured text.

[0217] If the result is "no customer service response", it means that the main interference during that period was environmental noise or non-human voice, and the customer service did not speak. This is considered "silent", and the structured text transcription reprocessing process will not be triggered.

[0218] It should be noted that the selection of the above-mentioned human voice features can be made adaptively according to the specific situation, and is not limited to the specific features shown in Table 5. This embodiment does not make specific limitations in this regard.

[0219] In one alternative approach, the short response recognition mechanism for single-channel call audio includes:

[0220] For target customer statements that still have a silence flag after performing dialogue flow logic verification, an audio segment within the first time interval is selected from the customer service audio as the context audio segment corresponding to the target customer statement, and the presence of preset human voice features is detected within the context audio segment. The first time interval is represented as [t1, t2 + T], where t1 represents the start time of the target customer statement, t2 represents the end time of the target customer statement, and T represents the silence threshold.

[0221] If the voice characteristics of a second person other than the customer are detected within the first time interval, the call audio is re-transcribed into structured text, and the above-mentioned silence detection method is performed again on the re-transcribed structured text; otherwise, the silence mark of the target customer's speech is maintained.

[0222] Specifically, the aforementioned voice characteristics can be derived from the voice interjection characteristics shown in Table 6.

[0223] Table 6

[0224]

[0225] Based on the characteristics of human voice insertion shown in Table 6, the presence of human voice insertion in mono audio is detected within the first time interval. The main feature dimensions detected are as follows:

[0226] Fundamental frequency presence detection: Using a multi-fundamental frequency tracking algorithm or harmonic product spectrum, the customer's primary fundamental frequency trajectory is first determined. Modeling is performed, and within the first time interval, a search is conducted to determine if any of the following exists: The second significant fundamental frequency peak, in addition to Calculate the minimum interval between two fundamental frequencies. If a second fundamental frequency is detected within the time window Furthermore, its frequency falls within the human voice range of 80-320Hz, and ΔF>20Hz, ensuring that it is not an overtone interference from the same sound source, and the duration of the second fundamental frequency is >100ms; at this time, it is determined that there are multiple human voices in the detection interval, and the result of the fundamental frequency detection is recorded as "satisfied", otherwise it is recorded as "not satisfied".

[0227] Formant Detection: Using cepstral analysis combined with a formant tracker, LPC analysis is performed on the audio frames to extract formant sequences F1, F2, and F3. These are compared to the center frequency distribution Pclient of the formants during normal speech to detect the presence of a new formant Pnew whose center frequency differs significantly from Pclient (e.g., a shift exceeding 100Hz), and whose energy is significantly higher than the background noise floor. If at least two independent formant groups are detected (Group A corresponding to the customer and Group B corresponding to the interjector), and the formants of Group B exhibit dynamic changes similar to human voice pronunciation over time, it is determined that multiple voices are present within the detection interval. In this case, the formant detection result is recorded as "satisfied"; otherwise, it is recorded as "unsatisfied".

[0228] For single-channel mixed audio, the judgment logic is more stringent because the channels cannot be physically separated. Only when the results of both the "fundamental frequency presence" and "formant detection" dimensions are "satisfied" will the first time interval be marked as "suspected customer service response" and the ASR reprocessing process be triggered; otherwise, it means that the main component during that time period is environmental noise or customer monologue, and it is judged as "silence is true".

[0229] Furthermore, within the second time interval [t2, t2+T], if the customer does not speak, it is only necessary to determine whether the customer service representative responds. At this time, the audio only contains the customer service representative's voice, and the aforementioned method for detecting human voice features corresponding to dual-channel audio can be used for detection.

[0230] Optionally, for sentences identified as "suspected customer service responses," the audio from the customer service channel during that period can be verified to confirm whether the customer service representative actually responded. Verification methods can include ASR transcription verification or manual verification, as long as verification is feasible. For example, ASR transcription verification is used here as an example for explanation, as follows:

[0231] Resubmit the audio from the customer service channel during the time period of the sentence described as a "suspected customer service response" to ASR for transcription. If reprocessing yields text, it indicates that customer service did respond, confirming that ASR missed transcribing it. Add the newly transcribed content to the conversation text record and remove the corresponding mute marker. If reprocessing still yields no text, mark it as "invalid audio, customer service did not provide a substantive response."

[0232] For example, a customer says, "This summer was the first time I turned on the air conditioner, but it didn't cool down for a long time," during which the customer service representative responds softly with "Hmm." Because this response is brief, it might be missed by the ASR (Automatic Responsive Response) engine. If it's dual-channel audio, the system analyzes the suspected silence period [t1, t2+T] from five dimensions: fundamental frequency, formants, duration, energy, and zero-crossing rate. If all these dimensions match the characteristics of a brief customer service response, it's marked as a "suspected customer service response," and ASR reprocessing is triggered. If it's single-channel audio, the system analyzes it from two dimensions: fundamental frequency presence and formant detection. If both dimensions match the characteristics of a human voice interruption, it's marked as a "suspected customer service response," and ASR reprocessing is triggered. The ASR engine successfully transcribes "Hmm," thus eliminating the suspicion of silence.

[0233] This embodiment improves the accuracy of silence detection results by performing short customer service response detection based on human voice characteristics on single-frequency and dual-frequency channel scenarios respectively.

[0234] Optionally, the above-mentioned silence detection method may also include: outputting the silence detection result.

[0235] The test results include the test conclusion, the sequence number of the silent sentence, and the reason for the silence, such as... Figure 5 As shown. Examples of attribute descriptions included in the detection results are as follows:

[0236] Compliance determination: If the entire call is not muted, the result is: Compliance.

[0237] Single violation determination: If no mute count is configured in the detection rules, a violation will be output if the entire call is muted once. Here, "prolonged silence without exemption intent" and "failure to respond within the time limit despite exemption" are both counted as one mute.

[0238] Prolonged silence without exemption intent. When a customer service representative fails to respond to a customer beyond the normal silence period without exemption intent, a violation will be marked on the "customer's message" where the representative did not respond. The silenced message number will be the customer's message, and the reason for silence will be "prolonged failure to respond to the customer."

[0239] Exemption granted but no response timed out. When a customer service representative intends to grant an exemption but fails to respond to the customer after the exemption inquiry period has expired, a violation will be marked on the sentence containing the exemption intention. The mute sentence number will be the sentence containing the exemption intention, and the violation reason will be "failure to respond to the customer after the exemption period has expired".

[0240] Cumulative count determination of violation: If the detection rule has a configured threshold N for the number of times the device has been muted (e.g., N=2), then:

[0241] When the number of times a call is muted exceeds the threshold N, the detection conclusion is output: violation, and the mute sentence number and mute reason of all mute events are output simultaneously.

[0242] When the total number of times the call is muted does not exceed the threshold N, the output conclusion is: compliant. However, the sequence number of the muted sentence and the reason for the muting event should still be output simultaneously.

[0243] The following provides a specific scenario for mute detection results. Assume the transcribed text structure of a call is as shown in Table 7, the system sets the standard mute threshold T=15 seconds, and the exemption time is uniformly 30 seconds.

[0244] Table 7

[0245]

[0246] The process for generating silence detection results is as follows:

[0247] (1) Analyze sentences 2 and 3.

[0248] Customer sentence 2 ends at 10:00:08. Customer service sentence 3 begins at 10:00:09. The waiting interval is 1 second, which is less than the standard mute threshold of 15 seconds, indicating that the customer service representative is not muted.

[0249] (2) Analyze sentence 3 and its following parts.

[0250] Sentence 3 contains the exemption intent keyword "Let me check for you." The system activates the exemption period, which begins at the end time of sentence 3 (10:00:11) and lasts for 30 seconds, ending at 10:00:41. During this period (10:00:11 to 10:00:41), customer messages (sentence 4) will not be muted. If customer service does not reply before 10:00:41, the exemption period will expire.

[0251] (3) Judgment.

[0252] Mark customer service sentence 3 as "Exemption granted but no response timed out". Record 1 instance of being muted. The mute sentence number is 3, and the reason for being mute is "no response to customer after the exemption time has expired".

[0253] (4) Analyze sentences 5 and 6.

[0254] Customer sentence 5 ends at 10:30:28. Customer service sentence 6 begins at 10:30:50. The waiting interval is 22 seconds, which is clearly greater than the usual threshold of 15 seconds. An examination revealed that the customer service representative did not speak between sentences 5 and 6 (no intention to waive this).

[0255] (5) Judgment.

[0256] Mark customer sentence 5 as "prolonged silence without exemption intent". Record 1 silence. The silence sentence number is 5, and the reason for silence is "no response to customer for a long time".

[0257] After obtaining the above silence test results, the following three different output formats can be used.

[0258] Scenario 1: The rule does not specify the number of times to mute (if it occurs, it is a violation).

[0259] If a call is marked as mute at least once, the conclusion is "violation". In this example, if two mute marks are marked, the output will list all mute events.

[0260] Output result: {

[0261] "Detection Conclusion": "Violation"

[0262] "Silence Incident": [

[0263] {

[0264] "Silent sentence number": 3,

[0265] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0266] },

[0267] {

[0268] "Silent sentence number": 5,

[0269] Reason for being silent: "Has not responded to customers for an extended period of time"

[0270] }

[0271] ]}.

[0272] Scenario 2: Rule configuration: Silence count threshold N=3 (cumulative violations).

[0273] A call is considered a "violation" if it contains at least three mute markers. In this example, two mute markers are present, which is compliant. The output should list all mute events.

[0274] {

[0275] "Test Conclusion": "Compliant"

[0276] "Silence Incident": [

[0277] {

[0278] "Silent sentence number": 3,

[0279] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0280] },

[0281] {

[0282] "Silent sentence number": 5,

[0283] Reason for being silent: "Has not responded to customers for an extended period of time"

[0284] }]}.

[0285] Scenario 3: Rule configuration: Silence threshold N=1 (cumulative violations).

[0286] The number of times the call was muted was 2, exceeding the threshold of 1, resulting in a "violation" conclusion, and a list of all muting events was provided.

[0287] Output result:

[0288] {

[0289] "Detection Conclusion": "Violation"

[0290] "Silence Incident": [

[0291] {

[0292] "Silent sentence number": 3,

[0293] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0294] },

[0295] {

[0296] "Silent sentence number": 5,

[0297] Reason for being silent: "Has not responded to customers for an extended period of time"

[0298] }

[0299] ]}.

[0300] At this point, the silence detection result is output, and the silence detection is complete.

[0301] It should be noted that the above settings for violation rules in the output results can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on them.

[0302] This invention provides a computer-readable storage medium including a stored program, wherein the program, when running, controls the device containing the computer-readable storage medium to perform the aforementioned silence detection method.

[0303] Specifically, the noise detection methods include:

[0304] Step S201: Obtain the call audio and perform structured text transcription on the call audio to obtain structured text.

[0305] Specifically, the aforementioned call audio includes voice data of conversations between customer service representatives and customers. Before transcribing the call audio into structured text, preprocessing can be performed on the call audio, such as filtering out unnecessary content for transcription, like music and noise.

[0306] In practice, Automatic Speech Recognition (ASR) can be used to process the call audio into structured text, resulting in structured text. It should be noted that the specific method of structured text transcription can be chosen according to requirements; this embodiment does not impose any specific limitations on this.

[0307] After transcribing the call audio into structured text, the structured text contains the customer service representative's statements, the customer's statements, and the timestamps corresponding to each statement. Below is an example of structured text, where `content` represents the dialogue content; `dialogIdIndex` represents the sentence ID; `startTime` is the start time of the dialogue content; `endTime` is the end time of the dialogue content; and `sender` is the role of the speaker (e.g., customer service representative or customer).

[0308] {

[0309] "content": "Hello, this is Gree Customer Service Center. How can I help you?"

[0310] "dialogIdIndex": "1",

[0311] "startTime": "2025-10-11 00:17:35",

[0312] "endTime": "2025-10-11 00:17:37",

[0313] "sender": "Customer Service",

[0314] },

[0315] {

[0316] "content": "Hello, do you offer free air conditioner cleaning now?"

[0317] "dialogIdIndex": "2",

[0318] "startTime": "2025-10-11 00:17:38",

[0319] "endTime": "2025-10-11 00:17:41",

[0320] "sender": "customer",

[0321] },

[0322] {

[0323] "content": "My air conditioner needs cleaning".

[0324] "dialogIdIndex": "3",

[0325] "startTime": "2025-10-11 00:17:41",

[0326] "endTime": "2025-10-11 00:17:43",

[0327] "sender": "customer",

[0328] },

[0329] {

[0330] "content": "Yes, is it a floor-standing or wall-mounted air conditioner?"

[0331] "dialogIdIndex": "4",

[0332] "startTime": "2025-10-11 00:17:49",

[0333] "endTime": "2025-10-11 00:17:50",

[0334] "sender": "Customer Service",

[0335] },

[0336] {

[0337] "content": "cabinet air conditioner",

[0338] "dialogIdIndex": "5",

[0339] "startTime": "2025-10-11 00:17:51",

[0340] "endTime": "2025-10-11 00:17:52",

[0341] "sender": "customer",

[0342] },

[0343] {

[0344] "content": "Okay, what is your address? We'll register you here."

[0345] "dialogIdIndex": "6",

[0346] "startTime": "2025-10-11 00:18:10",

[0347] "endTime": "2025-10-11 00:18:15",

[0348] "sender": "customer",

[0349] }

[0350] Step S202: Detect whether there is an intention to waive the requirement for additional processing time in the customer service statement. If there is an intention to waive the requirement, activate the waiver period of the preset duration starting from the end time of the customer service statement.

[0351] Exemption from intent detection is designed to address the issue of the basic silence detection rules being too simplistic and inflexible in customer service silence detection, leading to a large number of instances of customer service staff muting. This can improve the rationality of silence detection.

[0352] In actual customer service interactions, after a user speaks, customer service representatives may need to query information or work orders. It's necessary to allow them time for these queries to improve the system's operational rationality and fairness. Therefore, in step S202, exemption intent is detected in the customer service representative's statement. If exemption intent is detected, the exemption duration is determined.

[0353] Specifically, exemption intent detection methods such as regular expression matching, semantic similarity detection, or a combination of regular expression matching and semantic similarity matching can be used to identify exemption intent in customer service statements. In practice, regular expression matching should be prioritized due to its speed; once a match is found, the result is returned immediately. If no regular expression match is found, semantic matching is executed to ensure that non-standard expressions are also recognized.

[0354] It should be noted that the above rules for identifying exemption intent can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on this.

[0355] When a customer service representative's message is detected to have an intent to grant an exemption, an exemption period is activated. This exemption period has a duration TD, meaning it begins at the end time of the message and ends after TD. The exemption period applies only to the customer service representative; during this period, if a new customer message appears, the representative is allowed not to respond to it.

[0356] In practice, the exemption duration can be determined based on a pre-set exemption duration setting strategy.

[0357] It should be noted that the above-mentioned exemption duration setting strategy can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on it.

[0358] Step S203: For customer speech, if the start time of the customer speech is within the exemption period, skip the mute timer for the customer speech; if the start time of the customer speech is outside the exemption period, start the mute timer for the customer speech.

[0359] In step S202, the triggering mechanism for the exemption period has been determined. For customer service statements with an intention to grant an exemption, an exemption period is activated for each customer service statement.

[0360] Normally, when a customer speaks, the mute timer starts from the end time of their message. However, in the exemption mechanism described in this embodiment, if the customer's start time falls within a certain exemption period, customer service is allowed not to reply to the customer's message, i.e., the mute timer for that customer's message is skipped. After the exemption period ends, the mute timer starts again for new customer messages that occur after the end time of the exemption period.

[0361] Step S204: If the silence time obtained by the silence timer is greater than the preset silence threshold, or if customer service has not responded by the end of the exemption period, it is determined that a silence event has occurred.

[0362] It should be noted that the above-mentioned mute threshold can be set according to needs, such as 5 seconds, 10 seconds, etc., and this embodiment does not impose specific restrictions on it.

[0363] Through the above embodiments, call audio is acquired, and structured text transcription is performed on the call audio to obtain structured text. The structured text includes customer service statements, customer statements, and timestamps corresponding to each statement. The system detects whether the customer service statements contain an exemption intent indicating the need for additional processing time. If an exemption intent exists, a preset exemption period is activated starting from the end time of the customer service statement. For customer statements, if the start time of the customer statement is within the exemption period, the silence timer for the customer statement is skipped; if the start time of the customer statement is outside the exemption period, the silence timer for the customer statement is started. If the silence time obtained from the silence timer exceeds a preset silence threshold, or if the customer service representative has not responded by the end of the exemption period, a silence event is determined to exist. This application embodiment, through an exemption intent recognition mechanism, flexibly adjusts the silence detection threshold, effectively filters reasonable silence, and significantly improves the reliability and business adaptability of the detection results.

[0364] In one implementation, detecting whether a customer service representative's statement contains an intention to grant an exemption that requires additional processing time includes: performing regular expression matching on the customer service statement; if a preset exemption statement is matched, it is determined that the customer service representative's statement contains an intention to grant an exemption. This embodiment uses a regular expression matching strategy, which is fast and returns the result immediately upon successful matching, thereby improving the overall system response speed.

[0365] Specifically, for regular expression matching, a pre-defined regular expression can be used to quickly match customer service statements. This embodiment defines the following four exemption modes. If a match is successful, an exemption period is activated, and the specific exemption mode and the matched text fragment of the customer service statement are recorded. The exemption modes include:

[0366] (1) Query mode: Matches customer service representatives who clearly express their intention to query, verify, or confirm, such as "I'll check" or "I'll verify and see".

[0367] (2) Time estimation mode: Match customer service to inform customers of the waiting time, such as "wait a minute" or "it will take a little time".

[0368] (3) System status pattern: Match customer service to explain the waiting expression due to technical reasons such as system and network, such as "the system is loading" or "the network is a bit slow".

[0369] (4) Process description mode: Match the customer service description of the process being processed, such as "processing for you" or "I will help you operate it right away".

[0370] In one implementation, detecting whether there is an exemption intention in the customer service statement that indicates the need for additional processing time includes: performing semantic recognition on the customer service statement, and if a preset exemption semantic is identified, determining that there is an exemption intention in the customer service statement.

[0371] Specifically, this embodiment primarily employs a semantic similarity matching strategy. This strategy involves constructing an exemption intent semantic core library, encompassing various semantic cores related to queries, waiting, and processing. Next, text vectorization is performed, using a pre-trained language model to convert customer service statements and exemption intent core expressions into high-dimensional semantic vectors. Then, the cosine similarity between the customer service statement vector and each exemption intent core vector is calculated, and the highest similarity value is taken as the matching score. For example, a similarity threshold of 0.75 can be set; when the matching score exceeds this threshold, the customer service statement is deemed to contain an exemption intent.

[0372] In practice, regular expression matching can be used to quickly identify statements with obvious exemption intent. Then, a semantic similarity matching strategy can be used to further filter the remaining customer service statements whose meanings meet the exemption criteria but whose expression is not covered by the regular expression pattern. This ensures the accuracy of the exemption intent identification results. For example, the following is an example of exemption intent detection:

[0373] Customer service replied: "Please be patient, I'll check the system."

[0374] Regular expressions might not be able to match this colloquial expression. In this case, the system calculates semantic similarity and finds that the vector similarity between the sentence and the core expression of the exemption intent, "Please wait a moment, let me check the system," reaches 0.82, exceeding the threshold of 0.75. Therefore, the system successfully determines that the sentence contains an exemption intent and activates the exemption period.

[0375] In one implementation, timing the silence of a customer's speech includes: calculating the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer statement after the customer's speech, and using the time difference as the silence time.

[0376] The silence time can be expressed by the following formula:

[0377] ,

[0378] in, The end time for the customer's speech. The start time of the first customer service representative's statement after the customer's statement, or the first customer's statement. Indicates the duration of silence.

[0379] like >T, where T represents the preset mute threshold. This means that the customer service representative has not responded to this customer message for a long time. At this time, a mute mark is added to this customer message. The mute mark indicates that the customer service representative is suspected of muteing this customer message once.

[0380] like This indicates that customer service responded promptly or that the customer spoke continuously.

[0381] This embodiment uses the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer's statement after the customer's speech as the silence time, which reasonably describes the time range in which silence events may occur in actual dialogue scenarios, thereby enabling accurate identification of silence events.

[0382] In one implementation, the exemption period has an exemption duration, which is determined based on the complexity of the customer's problem and / or the customer service representative's attributes; wherein, the complexity of the customer's problem includes: simple problems, routine problems, and complex problems; and the customer service representative's attributes include: customer service representative level and customer service representative's proficiency in the business area.

[0383] In practice, the exemption period can be automatically adjusted based on the complexity of the customer's question and / or the customer service representative's attributes. Taking home appliance customer service as an example, common customer question types include pre-sales product inquiries, price and model comparisons, after-sales installation appointment inquiries, fault reporting, and complaint feedback. If the customer service representative is a senior pre-sales representative with many years of experience, the exemption period can be reduced according to the preset adjustment strategy when a customer inquires about pre-sales issues; if the customer service representative is a new after-sales representative, the exemption period should be extended when a customer inquires about pre-sales issues. If the customer service representative is only making a simple inquiry, the exemption period should be reduced; if complex operations are required, a longer exemption period should be granted.

[0384] To adapt to the varying management granularity requirements of different business scenarios, this embodiment provides multiple configurable exemption duration setting strategies. These are illustrated below through three specific scenario examples.

[0385] Example scenario 1: Setting an exemption duration strategy based on the complexity of the customer's problem.

[0386] This exemption duration setting strategy is suitable for business scenarios that prioritize simple, stable, and easily manageable rules. This strategy sets a fixed exemption duration based solely on the estimated complexity of the customer's problem, without considering individual customer service differences. Please refer to Table 1, which schematically illustrates one exemption duration setting strategy.

[0387] Table 1

[0388]

[0389] Table 1 categorizes issues into simple queries, routine processing, and complex operations based on their complexity. Table 1 also shows typical scenario descriptions, exemption durations, and specific operations that customer service representatives need to perform for different issue types.

[0390] Taking the exemption time setting strategy shown in Table 1 as an example, if a customer asks, "I lost the remote control for my air conditioner, how much does it cost to replace it?", the system recognizes this as a simple query and automatically sets a 30-second exemption time. Customer service needs to check the price of the replacement part and reply within this time. The advantage of this exemption time setting strategy is that the rules are transparent and the computational overhead is minimal.

[0391] Scenario Example 2: Fixed time settings based on customer service attributes.

[0392] This exemption duration setting strategy is suitable for scenarios that emphasize personalized management and fairness, have a highly homogeneous business type, and exhibit minimal differences in issue complexity. The exemption duration is primarily set based on the customer service personnel's own attributes. Its advantage lies in its extremely simple rules, but it does not consider differences in issue complexity. Please refer to Table 2, which schematically illustrates one exemption duration setting strategy.

[0393] Table 2

[0394]

[0395] Table 2 shows that the adjustment factors for the exemption duration are related to the customer service level and the matching degree of the customer service's business area. When calculating the exemption duration, both the customer service level and the matching degree of the customer service's business area need to be considered. Table 2 shows the specific exemption duration for different customer service levels and different business area matching degrees, as well as the relevant explanations for setting the exemption duration.

[0396] The calculation formula for the exemption duration is as follows: Exemption duration = Exemption duration corresponding to customer service level + Adjustment time corresponding to business area matching.

[0397] Taking a junior customer service representative handling a cross-domain issue as an example, the exemption time for junior customer service representatives is 40 seconds, and the exemption time for business domain matching is 15 seconds. Therefore, the final exemption time is: 0 seconds + 15 seconds = 55 seconds.

[0398] Scenario Example 3: Weighted settings for comprehensive question types and customer service attributes.

[0399] This embodiment, based on distinguishing problem types, further considers the differences in customer service personnel capabilities to set a baseline exemption duration and adjustment coefficient, making the exemption duration more reasonable while adapting to business needs. Please refer to Table 3, which schematically illustrates an exemption duration setting strategy for this scenario.

[0400] Table 3

[0401]

[0402] Table 3 shows that the adjustment factors for the exemption duration are related to the problem type, customer service level, and the degree of matching between the customer service representative's business area and the exemption duration. When calculating the exemption duration, it is necessary to consider the problem type, customer service level, and the degree of matching between the customer service representative's business area and the exemption duration simultaneously. Table 3 shows the specific exemption duration for different problem types, different customer service levels, and different degrees of matching between business areas, as well as the relevant explanations for setting the exemption duration.

[0403] The calculation formula for the exemption duration is as follows: Exemption duration = Exemption duration corresponding to the problem type × Customer service level coefficient + Business area matching adjustment time.

[0404] Taking a senior customer service representative handling a complex cross-domain operational issue as an example, the exemption time corresponding to the issue type is 90 seconds, the senior customer service representative's coefficient is 0.9, and the business domain matching adjustment time is increased by 15 seconds, resulting in a final exemption time of 90 seconds. 0.9 + 15 seconds = 96 seconds.

[0405] By flexibly setting the exemption duration, it is possible to fully adapt to the exemption requirements of different business scenarios, thereby improving the accuracy of the overall silence detection method.

[0406] It should be noted that the exemption duration setting strategy in this embodiment can be set according to needs and is not limited to the three exemption duration setting strategies mentioned above.

[0407] In one implementation, an exemption status management mechanism can also be set up. Specifically, if a new customer service message is detected during the exemption period, the exemption period is terminated.

[0408] Specifically, the exemption status management mechanism also includes: if a new customer service message contains an intention to grant an exemption, a new exemption period will be reactivated starting from the end time of the new customer service message.

[0409] For example, such as Figure 3 As shown, the process of the exemption status management mechanism is as follows:

[0410] Step S301: When an exemption intent is detected, the exemption period is activated starting from the end of the current customer service speech.

[0411] Step S302: During the exemption period, no mute timer is set for new customer service messages.

[0412] Step S303: If the customer service representative responds to the customer within the exemption period, the exemption period ends immediately.

[0413] Step S304: If the exemption period expires and customer service still does not respond, the exemption status is terminated and it is immediately determined that a mute event has occurred.

[0414] Step S305: Output the judgment result of the silence detection.

[0415] For example, the following uses "Air Conditioner Repair Progress Inquiry" as an example to illustrate... Figure 3 The exemption status management mechanism is explained in detail. Sentences 3 to 5 in the "Air Conditioner Repair Progress Inquiry" section are as follows:

[0416] Sentence 3: The customer service representative said, "Let me check the progress of your work order; it will take a little time."

[0417] Sentence 4: The customer says, "Okay, also, my air conditioner is making strange noises."

[0418] Sentence 5: Customer service representative's message: "Found it, your work order is being processed. Regarding the abnormal noise issue you just mentioned, I need to understand it in more detail."

[0419] Sentence 3 is a customer service statement, so the first step is to check if sentence 3 contains an intention to grant an exemption. Since the keywords "check it out" and "it will take a little time" were detected, it can be determined that sentence 3 contains an intention to grant an exemption. Taking the above scenario example 3 as an example, combining the problem type (routine processing, 60 seconds) and the customer service level (intermediate, coefficient 1.0), and setting the exemption time to 60 seconds, the exemption period is represented as follows: , This indicates the end time of sentence 3, in seconds.

[0420] Next, regarding sentence 4, assuming that sentence 4 occurs within the exemption period, then silence detection will not be performed on sentence 4 during the exemption period.

[0421] Finally, regarding sentence 5, assuming that the customer service representative replied 45 seconds after the start of the exemption period, then the customer service representative's reply was within the exemption period, and the exemption status terminated.

[0422] This embodiment sets up a complete management mechanism of "activation → termination → reactivation" for the exemption period, which fully adapts to all possible situations during the exemption period, improves the flexibility and rationality of the exemption period mechanism, and thus improves the accuracy of the silence detection results.

[0423] In one embodiment, the above-mentioned silence detection method further includes: if the silence time obtained by the silence timing is greater than the silence threshold, marking the customer's speech with a silence mark; if the customer service representative still has not responded by the end of the exemption period, marking the customer service representative's speech corresponding to the exemption period with a silence mark. This embodiment achieves accurate positioning of different silence events by adding silence marks, which facilitates subsequent review of the silence detection results.

[0424] In one embodiment, the above-mentioned silence detection method further includes: taking a customer's speech with a silence mark as the target customer's speech, performing a dialogue flow logic check on the target customer's speech, and detecting whether the previous sentence of the target customer's speech was a customer service speech; if the previous sentence of the target customer's speech was a customer service speech, and the end time of the target customer's speech is earlier than the end time of the previous customer service speech, canceling the silence mark of the target customer's speech.

[0425] In practice, user interruptions may occur, leading to disordered sentence order in the structured text. This is because in actual conversations, the same speaker may pause naturally, or one party may interrupt the other. An ideal Automatic Speech Recognition (ASR) system should be able to segment sentences containing long pauses appropriately and accurately present interaction sequences such as "customer service - customer (interruption) - customer service". However, ASR models often suffer from the following two types of problems due to coarse or incorrect sentence segmentation granularity:

[0426] 1. Incorrectly merging spoken content from the same person, including natural pauses, into a single text;

[0427] 2. Cover up other people's interruptions within the same speaker's text block.

[0428] Both of these errors distort the original temporal logic of the dialogue, leading to misjudgments in silence detection.

[0429] Therefore, this embodiment proposes a dialogue flow logic verification mechanism. Its core principle lies in using the precise start and end timestamps of each sentence in the structured text to detect timing inconsistencies, thereby inferring potential merging or segmentation errors in ASR transcription. The specific verification rule is as follows: when a silence marker is detected for a customer's statement, if the preceding sentence is a customer service statement, and the end time of this customer's statement is earlier than the end time of the previous customer service statement (customer interruption), then it indicates an ASR transcription anomaly or customer interruption. In this case, the system will rule out the possibility of customer service silence and remove the silence marker from that customer's statement.

[0430] For example, as shown in Table 4, Table 4 presents a structured text of a conversation between a customer service representative and a customer to verify the delivery address.

[0431] Table 4

[0432]

[0433] In response to this dialogue, execute as follows Figure 4 The silence detection process shown, which includes a dialogue flow logic verification mechanism, includes the following steps:

[0434] Step S401, Initial silence detection.

[0435] For customer sentence 5 (content "201") in Table 4, calculate the interval between its end time and the start time of the next sentence (customer sentence 6): ΔT = t_start(sentence 6) - t_end(sentence 5) = 00:17:55 - 00:17:42 = 13 seconds.

[0436] Assuming the silence threshold T = 5 seconds, since 13 seconds > 5 seconds, the system initially flags the customer service representative for a single instance of silencing the customer in sentence 5.

[0437] Step S402: Trigger dialogue flow logic verification.

[0438] The system applies validation rules to check the contextual timing of the client's sentence 5:

[0439] Check if the sentence preceding customer sentence 5 (sentence 4) was spoken by customer service. The result is yes.

[0440] Check if the end time of customer sentence 5 (00:17:42) is earlier than the end time of the previous customer service statement (sentence 4) (00:17:52). Since 00:17:42 < 00:17:52, the result is yes.

[0441] Step S403, Analysis and Judgment.

[0442] The aforementioned temporal inconsistency (the customer finished speaking before the customer service representative finished speaking) reveals that the actual dialogue process should have been as follows: after the customer service representative repeated the address "Building 5," the customer immediately interrupted to confirm "201," and then the customer service representative continued with the subsequent remarks. The ASR system incorrectly merged the two segments of the customer service representative's speech before and after the interruption into sentence 4, thus concealing the fact of the interruption and causing sentence 5 to be misjudged as an independent, unanswered customer question.

[0443] Step S404: Perform the correction.

[0444] Based on the dialogue flow verification results, the system determined that this silence marker was caused by an ASR transcription anomaly (the interruption was not correctly segmented), and not by a customer service error. Therefore, the silence marker for customer sentence 5 was removed, and the silence detection result was corrected.

[0445] Through this mechanism, this embodiment can effectively identify and correct mute misjudgments caused by ASR sentence merging, interruption and segmentation errors, etc., and significantly improve the accuracy and reliability of mute detection in complex dialogue interaction scenarios.

[0446] In one implementation, the above-mentioned silence detection method also includes a short response recognition mechanism. This mechanism addresses silence situations caused by short customer service responses being missed during ASR transcription. Specifically, during actual conversations, while the customer is describing their situation, the customer service representative is often in a state of reassuring or listening to the customer, and frequently provides short responses. These short responses are easily missed during ASR transcription, and thus, during text-based detection, the customer service representative may be mistakenly judged as violating the silence rule.

[0447] To address this issue, a brief response recognition mechanism can be adopted, which includes: after performing dialogue flow logic verification, for the target customer's speech that still has a silence mark, detecting whether there are preset customer service voice features in the context audio segment corresponding to the target customer's speech in the customer service audio; if customer service voice features are detected in the context audio segment, canceling the silence mark of the target customer's speech, re-transing the customer service audio into structured text, and re-executing the silence detection method on the re-transcribed structured text.

[0448] In practice, call audio is handled in two ways. The first is dual-channel, which includes two independent audio streams: customer audio and customer service audio. The customer audio only contains all the sounds from the customer service side. The second is single-channel, which has only one mixed audio stream, where the customer service and customer voices are recorded together. The audio includes all the sounds from both sides.

[0449] For example, for dual-channel call audio, a short response recognition mechanism includes:

[0450] For target customer statements that still retain a silence flag after the dialogue flow logic verification, an audio segment within the first time interval is selected from the customer service audio as the context audio segment corresponding to the target customer statement. The presence of preset human voice features is then detected within this context audio segment. The first time interval is represented as [t1, t2 + T], where t1 represents the start time of the target customer service statement, t2 represents the end time of the target customer service statement, and T represents the silence threshold. If preset human voice features are detected in the context audio segment, the customer service audio is re-transcribed into structured text, and the aforementioned silence detection method is repeated on the re-transcribed structured text; otherwise, the silence flag for the target customer statement is maintained.

[0451] Specifically, the aforementioned voice features refer to customer service voice response features. Customer service voice responses typically contain specific short customer service phrases. Therefore, customer service voice response features can be constructed based on this characteristic to distinguish them from environmental noise features in call audio. For example, the specific short customer service phrase features shown in Table 5 can be used.

[0452] Table 5

[0453]

[0454]

[0455] Based on the customer service voice response features described in Table 5, five feature dimensions of the customer service speech are detected in the first time interval [t1, t2+T], including:

[0456] Duration detection: A short-time energy thresholding method is used. First, the short-time energy of audio frames within a first time interval is calculated. A noise floor threshold is set (e.g., -40dB), and continuous speech segments with frequencies above the noise floor threshold are extracted. The total duration of these continuous speech segments is then calculated. If it falls within the duration range of "customer service short phrase characteristics": 120ms≤ If the duration is ≤700ms, the duration detection result is recorded as "satisfied"; otherwise, it is recorded as "not satisfied".

[0457] Fundamental frequency detection: Fundamental frequency estimation is performed using the autocorrelation function method or the YIN algorithm. If a valid fundamental frequency is detected... It falls within the male (80-180Hz) or female (160-320Hz) range, and the fundamental frequency change rate within the segment is... If the frequency is less than the preset threshold, it indicates a constant frequency of non-mechanical noise, and the result is recorded as "satisfied" for the fundamental frequency detection; otherwise, it is recorded as "not satisfied".

[0458] Energy detection: The energy spectrum is calculated using short-time Fourier transform, and the energy envelope is extracted. The difference between the maximum energy Em within a segment and the background noise E is calculated. .like If the value is greater than 6dB and the variance σE of the energy envelope is greater than the preset threshold, it indicates fluctuations and non-stationary noise. In this case, the energy detection result is recorded as "satisfied"; otherwise, it is recorded as "unsatisfied".

[0459] Zero-crossing rate detection: Directly count the number of zero-crossing points in the audio frame and calculate the average value ZCRavg per unit time. If ZCRavg falls in the voiced area or transition area, or shows obvious segmented changes, such as high at the beginning and low at the end, and there are no constant extreme values ​​across the entire frequency band, such as the constant median of white noise, then the zero-crossing rate detection result is recorded as "satisfied". Otherwise, it is recorded as "not satisfied".

[0460] Formant detection: Linear predictive coding (LPC) analysis is used to extract the frequencies and bandwidths of the first 4-5 formants. If at least 2 significant formant peaks are detected, and their frequency positions conform to the typical distribution of human vowels or nasal sounds, such as F1 between 300-1000Hz, and the formant frequency changes smoothly over time, then the formant detection result is recorded as "satisfied"; otherwise, it is recorded as "not satisfied".

[0461] After the testing is completed, a comprehensive judgment is made, and the test results of the above 5 dimensions are weighted and scored. Either of the following two methods can be used:

[0462] (1) Full match mode: If the detection results of all 5 dimensions are marked as "satisfied", then [t1,t2+T] is directly determined as "suspected customer service response". At this time, the structured text transcription process needs to be re-executed, and the above-mentioned silence detection method is executed again on the re-transcribed structured text.

[0463] (2) Weighted threshold mode: Considering the complexity of the actual environment, a certain degree of fault tolerance is allowed. The system sets a weight for each dimension (e.g., fundamental frequency 0.3, formant 0.25, duration 0.20, energy 0.15, zero crossing rate 0.10). If the number of feature dimensions is ≥4, or the total weighted score is ≥0.8 (out of 1.0), then [t1, t2+T] is determined to be "suspected customer service response". At this time, the structured text transcription process needs to be repeated, and the above-mentioned silence detection method is executed again on the re-transcribed structured text.

[0464] If the result is "no customer service response", it means that the main interference during that period was environmental noise or non-human voice, and the customer service did not speak. This is considered "silent", and the structured text transcription reprocessing process will not be triggered.

[0465] It should be noted that the selection of the above-mentioned human voice features can be made adaptively according to the specific situation, and is not limited to the specific features shown in Table 5. This embodiment does not make specific limitations in this regard.

[0466] In one alternative approach, the short response recognition mechanism for single-channel call audio includes:

[0467] For target customer statements that still have a silence flag after performing dialogue flow logic verification, an audio segment within the first time interval is selected from the customer service audio as the context audio segment corresponding to the target customer statement, and the presence of preset human voice features is detected within the context audio segment. The first time interval is represented as [t1, t2 + T], where t1 represents the start time of the target customer statement, t2 represents the end time of the target customer statement, and T represents the silence threshold.

[0468] If the voice characteristics of a second person other than the customer are detected within the first time interval, the call audio is re-transcribed into structured text, and the above-mentioned silence detection method is performed again on the re-transcribed structured text; otherwise, the silence mark of the target customer's speech is maintained.

[0469] Specifically, the aforementioned voice characteristics can be derived from the voice interjection characteristics shown in Table 6.

[0470] Table 6

[0471]

[0472] Based on the characteristics of human voice insertion shown in Table 6, the presence of human voice insertion in mono audio is detected within the first time interval. The main feature dimensions detected are as follows:

[0473] Fundamental frequency presence detection: Using a multi-fundamental frequency tracking algorithm or harmonic product spectrum, the customer's primary fundamental frequency trajectory is first determined. Modeling is performed, and within the first time interval, a search is conducted to determine if any of the following exists: The second significant fundamental frequency peak, in addition to Calculate the minimum interval between two fundamental frequencies. If a second fundamental frequency is detected within the time window Furthermore, its frequency falls within the human voice range of 80-320Hz, and ΔF>20Hz, ensuring that it is not an overtone interference from the same sound source, and the duration of the second fundamental frequency is >100ms; at this time, it is determined that there are multiple human voices in the detection interval, and the result of the fundamental frequency detection is recorded as "satisfied", otherwise it is recorded as "not satisfied".

[0474] Formant Detection: Using cepstral analysis combined with a formant tracker, LPC analysis is performed on the audio frames to extract formant sequences F1, F2, and F3. These are compared to the center frequency distribution Pclient of the formants during normal speech to detect the presence of a new formant Pnew whose center frequency differs significantly from Pclient (e.g., a shift exceeding 100Hz), and whose energy is significantly higher than the background noise floor. If at least two independent formant groups are detected (Group A corresponding to the customer and Group B corresponding to the interjector), and the formants of Group B exhibit dynamic changes similar to human voice pronunciation over time, it is determined that multiple voices are present within the detection interval. In this case, the formant detection result is recorded as "satisfied"; otherwise, it is recorded as "unsatisfied".

[0475] For single-channel mixed audio, the judgment logic is more stringent because the channels cannot be physically separated. Only when the results of both the "fundamental frequency presence" and "formant detection" dimensions are "satisfied" will the first time interval be marked as "suspected customer service response" and the ASR reprocessing process be triggered; otherwise, it means that the main component during that time period is environmental noise or customer monologue, and it is judged as "silence is true".

[0476] Furthermore, within the second time interval [t2, t2+T], if the customer does not speak, it is only necessary to determine whether the customer service representative responds. At this time, the audio only contains the customer service representative's voice, and the aforementioned method for detecting human voice features corresponding to dual-channel audio can be used for detection.

[0477] Optionally, for sentences identified as "suspected customer service responses," the audio from the customer service channel during that period can be verified to confirm whether the customer service representative actually responded. Verification methods can include ASR transcription verification or manual verification, as long as verification is feasible. For example, ASR transcription verification is used here as an example for explanation, as follows:

[0478] Resubmit the audio from the customer service channel during the time period of the sentence described as a "suspected customer service response" to ASR for transcription. If reprocessing yields text, it indicates that customer service did respond, confirming that ASR missed transcribing it. Add the newly transcribed content to the conversation text record and remove the corresponding mute marker. If reprocessing still yields no text, mark it as "invalid audio, customer service did not provide a substantive response."

[0479] For example, a customer says, "This summer was the first time I turned on the air conditioner, but it didn't cool down for a long time," during which the customer service representative responds softly with "Hmm." Because this response is brief, it might be missed by the ASR (Automatic Responsive Response) engine. If it's dual-channel audio, the system analyzes the suspected silence period [t1, t2+T] from five dimensions: fundamental frequency, formants, duration, energy, and zero-crossing rate. If all these dimensions match the characteristics of a brief customer service response, it's marked as a "suspected customer service response," and ASR reprocessing is triggered. If it's single-channel audio, the system analyzes it from two dimensions: fundamental frequency presence and formant detection. If both dimensions match the characteristics of a human voice interruption, it's marked as a "suspected customer service response," and ASR reprocessing is triggered. The ASR engine successfully transcribes "Hmm," thus eliminating the suspicion of silence.

[0480] This embodiment improves the accuracy of silence detection results by performing short customer service response detection based on human voice characteristics on single-frequency and dual-frequency channel scenarios respectively.

[0481] Optionally, the above-mentioned silence detection method may also include: outputting the silence detection result.

[0482] The test results include the test conclusion, the sequence number of the silent sentence, and the reason for the silence, such as... Figure 5 As shown. Examples of attribute descriptions included in the detection results are as follows:

[0483] Compliance determination: If the entire call is not muted, the result is: Compliance.

[0484] Single violation determination: If no mute count is configured in the detection rules, a violation will be output if the entire call is muted once. Here, "prolonged silence without exemption intent" and "failure to respond within the time limit despite exemption" are both counted as one mute.

[0485] Prolonged silence without exemption intent. When a customer service representative fails to respond to a customer beyond the normal silence period without exemption intent, a violation will be marked on the "customer's message" where the representative did not respond. The silenced message number will be the customer's message, and the reason for silence will be "prolonged failure to respond to the customer."

[0486] Exemption granted but no response timed out. When a customer service representative intends to grant an exemption but fails to respond to the customer after the exemption inquiry period has expired, a violation will be marked on the sentence containing the exemption intention. The mute sentence number will be the sentence containing the exemption intention, and the violation reason will be "failure to respond to the customer after the exemption period has expired".

[0487] Cumulative count determination of violation: If the detection rule has a configured threshold N for the number of times the device has been muted (e.g., N=2), then:

[0488] When the number of times a call is muted exceeds the threshold N, the detection conclusion is output: violation, and the mute sentence number and mute reason of all mute events are output simultaneously.

[0489] When the total number of times the call is muted does not exceed the threshold N, the output conclusion is: compliant. However, the sequence number of the muted sentence and the reason for the muting event should still be output simultaneously.

[0490] The following provides a specific scenario for mute detection results. Assume the transcribed text structure of a call is as shown in Table 7, the system sets the standard mute threshold T=15 seconds, and the exemption time is uniformly 30 seconds.

[0491] Table 7

[0492]

[0493] The process for generating silence detection results is as follows:

[0494] (1) Analyze sentences 2 and 3.

[0495] Customer sentence 2 ends at 10:00:08. Customer service sentence 3 begins at 10:00:09. The waiting interval is 1 second, which is less than the standard mute threshold of 15 seconds, indicating that the customer service representative is not muted.

[0496] (2) Analyze sentence 3 and its following parts.

[0497] Sentence 3 contains the exemption intent keyword "Let me check for you." The system activates the exemption period, which begins at the end time of sentence 3 (10:00:11) and lasts for 30 seconds, ending at 10:00:41. During this period (10:00:11 to 10:00:41), customer messages (sentence 4) will not be muted. If customer service does not reply before 10:00:41, the exemption period will expire.

[0498] (3) Judgment.

[0499] Mark customer service sentence 3 as "Exemption granted but no response timed out". Record 1 instance of being muted. The mute sentence number is 3, and the reason for being mute is "no response to customer after the exemption time has expired".

[0500] (4) Analyze sentences 5 and 6.

[0501] Customer sentence 5 ends at 10:30:28. Customer service sentence 6 begins at 10:30:50. The waiting interval is 22 seconds, which is clearly greater than the usual threshold of 15 seconds. An examination revealed that the customer service representative did not speak between sentences 5 and 6 (no intention to waive this).

[0502] (5) Judgment.

[0503] Mark customer sentence 5 as "prolonged silence without exemption intent". Record 1 silence. The silence sentence number is 5, and the reason for silence is "no response to customer for a long time".

[0504] After obtaining the above silence test results, the following three different output formats can be used.

[0505] Scenario 1: The rule does not specify the number of times to mute (if it occurs, it is a violation).

[0506] If a call is marked as mute at least once, the conclusion is "violation". In this example, if two mute marks are marked, the output will list all mute events.

[0507] Output result: {

[0508] "Detection Conclusion": "Violation"

[0509] "Silence Incident": [

[0510] {

[0511] "Silent sentence number": 3,

[0512] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0513] },

[0514] {

[0515] "Silent sentence number": 5,

[0516] Reason for being silent: "Has not responded to customers for an extended period of time"

[0517] }

[0518] ]}.

[0519] Scenario 2: Rule configuration: Silence count threshold N=3 (cumulative violations).

[0520] A call is considered a "violation" if it contains at least three mute markers. In this example, two mute markers are present, which is compliant. The output should list all mute events.

[0521] {

[0522] "Test Conclusion": "Compliant"

[0523] "Silence Incident": [

[0524] {

[0525] "Silent sentence number": 3,

[0526] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0527] },

[0528] {

[0529] "Silent sentence number": 5,

[0530] Reason for being silent: "Has not responded to customers for an extended period of time"

[0531] }]}.

[0532] Scenario 3: Rule configuration: Silence threshold N=1 (cumulative violations).

[0533] The number of times the call was muted was 2, exceeding the threshold of 1, resulting in a "violation" conclusion, and a list of all muting events was provided.

[0534] Output result:

[0535] {

[0536] "Detection Conclusion": "Violation"

[0537] "Silence Incident": [

[0538] {

[0539] "Silent sentence number": 3,

[0540] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0541] },

[0542] {

[0543] "Silent sentence number": 5,

[0544] Reason for being silent: "Has not responded to customers for an extended period of time"

[0545] }

[0546] ]}.

[0547] At this point, the silence detection result is output, and the silence detection is complete.

[0548] It should be noted that the above settings for violation rules in the output results can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on them.

[0549] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0550] Step S201: Obtain the call audio and perform structured text transcription on the call audio to obtain structured text.

[0551] Specifically, the aforementioned call audio includes voice data of conversations between customer service representatives and customers. Before transcribing the call audio into structured text, preprocessing can be performed on the call audio, such as filtering out unnecessary content for transcription, like music and noise.

[0552] In practice, Automatic Speech Recognition (ASR) can be used to process the call audio into structured text, resulting in structured text. It should be noted that the specific method of structured text transcription can be chosen according to requirements; this embodiment does not impose any specific limitations on this.

[0553] After transcribing the call audio into structured text, the structured text contains the customer service representative's statements, the customer's statements, and the timestamps corresponding to each statement. Below is an example of structured text, where `content` represents the dialogue content; `dialogIdIndex` represents the sentence ID; `startTime` is the start time of the dialogue content; `endTime` is the end time of the dialogue content; and `sender` is the role of the speaker (e.g., customer service representative or customer).

[0554] {

[0555] "content": "Hello, this is Gree Customer Service Center. How can I help you?"

[0556] "dialogIdIndex": "1",

[0557] "startTime": "2025-10-11 00:17:35",

[0558] "endTime": "2025-10-11 00:17:37",

[0559] "sender": "Customer Service",

[0560] },

[0561] {

[0562] "content": "Hello, do you offer free air conditioner cleaning now?"

[0563] "dialogIdIndex": "2",

[0564] "startTime": "2025-10-11 00:17:38",

[0565] "endTime": "2025-10-11 00:17:41",

[0566] "sender": "customer",

[0567] },

[0568] {

[0569] "content": "My air conditioner needs cleaning".

[0570] "dialogIdIndex": "3",

[0571] "startTime": "2025-10-11 00:17:41",

[0572] "endTime": "2025-10-11 00:17:43",

[0573] "sender": "customer",

[0574] },

[0575] {

[0576] "content": "Yes, is it a floor-standing or wall-mounted air conditioner?"

[0577] "dialogIdIndex": "4",

[0578] "startTime": "2025-10-11 00:17:49",

[0579] "endTime": "2025-10-11 00:17:50",

[0580] "sender": "Customer Service",

[0581] },

[0582] {

[0583] "content": "cabinet air conditioner",

[0584] "dialogIdIndex": "5",

[0585] "startTime": "2025-10-11 00:17:51",

[0586] "endTime": "2025-10-11 00:17:52",

[0587] "sender": "customer",

[0588] },

[0589] {

[0590] "content": "Okay, what is your address? We'll register you here."

[0591] "dialogIdIndex": "6",

[0592] "startTime": "2025-10-11 00:18:10",

[0593] "endTime": "2025-10-11 00:18:15",

[0594] "sender": "customer",

[0595] }

[0596] Step S202: Detect whether there is an intention to waive the requirement for additional processing time in the customer service statement. If there is an intention to waive the requirement, activate the waiver period of the preset duration starting from the end time of the customer service statement.

[0597] Exemption from intent detection is designed to address the issue of the basic silence detection rules being too simplistic and inflexible in customer service silence detection, leading to a large number of instances of customer service staff muting. This can improve the rationality of silence detection.

[0598] In actual customer service interactions, after a user speaks, customer service representatives may need to query information or work orders. It's necessary to allow them time for these queries to improve the system's operational rationality and fairness. Therefore, in step S202, exemption intent is detected in the customer service representative's statement. If exemption intent is detected, the exemption duration is determined.

[0599] Specifically, exemption intent detection methods such as regular expression matching, semantic similarity detection, or a combination of regular expression matching and semantic similarity matching can be used to identify exemption intent in customer service statements. In practice, regular expression matching should be prioritized due to its speed; once a match is found, the result is returned immediately. If no regular expression match is found, semantic matching is executed to ensure that non-standard expressions are also recognized.

[0600] It should be noted that the above rules for identifying exemption intent can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on this.

[0601] When a customer service representative's message is detected to have an intent to grant an exemption, an exemption period is activated. This exemption period has a duration TD, meaning it begins at the end time of the message and ends after TD. The exemption period applies only to the customer service representative; during this period, if a new customer message appears, the representative is allowed not to respond to it.

[0602] In practice, the exemption duration can be determined based on a pre-set exemption duration setting strategy.

[0603] It should be noted that the above-mentioned exemption duration setting strategy can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on it.

[0604] Step S203: For customer speech, if the start time of the customer speech is within the exemption period, skip the mute timer for the customer speech; if the start time of the customer speech is outside the exemption period, start the mute timer for the customer speech.

[0605] In step S202, the triggering mechanism for the exemption period has been determined. For customer service statements with an intention to grant an exemption, an exemption period is activated for each customer service statement.

[0606] Normally, when a customer speaks, the mute timer starts from the end time of their message. However, in the exemption mechanism described in this embodiment, if the customer's start time falls within a certain exemption period, customer service is allowed not to reply to the customer's message, i.e., the mute timer for that customer's message is skipped. After the exemption period ends, the mute timer starts again for new customer messages that occur after the end time of the exemption period.

[0607] Step S204: If the silence time obtained by the silence timer is greater than the preset silence threshold, or if customer service has not responded by the end of the exemption period, it is determined that a silence event has occurred.

[0608] It should be noted that the above-mentioned mute threshold can be set according to needs, such as 5 seconds, 10 seconds, etc., and this embodiment does not impose specific restrictions on it.

[0609] Through the above embodiments, call audio is acquired, and structured text transcription is performed on the call audio to obtain structured text. The structured text includes customer service statements, customer statements, and timestamps corresponding to each statement. The system detects whether the customer service statements contain an exemption intent indicating the need for additional processing time. If an exemption intent exists, a preset exemption period is activated starting from the end time of the customer service statement. For customer statements, if the start time of the customer statement is within the exemption period, the silence timer for the customer statement is skipped; if the start time of the customer statement is outside the exemption period, the silence timer for the customer statement is started. If the silence time obtained from the silence timer exceeds a preset silence threshold, or if the customer service representative has not responded by the end of the exemption period, a silence event is determined to exist. This application embodiment, through an exemption intent recognition mechanism, flexibly adjusts the silence detection threshold, effectively filters reasonable silence, and significantly improves the reliability and business adaptability of the detection results.

[0610] In one implementation, detecting whether a customer service representative's statement contains an intention to grant an exemption that requires additional processing time includes: performing regular expression matching on the customer service statement; if a preset exemption statement is matched, it is determined that the customer service representative's statement contains an intention to grant an exemption. This embodiment uses a regular expression matching strategy, which is fast and returns the result immediately upon successful matching, thereby improving the overall system response speed.

[0611] Specifically, for regular expression matching, a pre-defined regular expression can be used to quickly match customer service statements. This embodiment defines the following four exemption modes. If a match is successful, an exemption period is activated, and the specific exemption mode and the matched text fragment of the customer service statement are recorded. The exemption modes include:

[0612] (1) Query mode: Matches customer service representatives who clearly express their intention to query, verify, or confirm, such as "I'll check" or "I'll verify and see".

[0613] (2) Time estimation mode: Match customer service to inform customers of the waiting time, such as "wait a minute" or "it will take a little time".

[0614] (3) System status pattern: Match customer service to explain the waiting expression due to technical reasons such as system and network, such as "the system is loading" or "the network is a bit slow".

[0615] (4) Process description mode: Match the customer service description of the process being processed, such as "processing for you" or "I will help you operate it right away".

[0616] In one implementation, detecting whether there is an exemption intention in the customer service statement that indicates the need for additional processing time includes: performing semantic recognition on the customer service statement, and if a preset exemption semantic is identified, determining that there is an exemption intention in the customer service statement.

[0617] Specifically, this embodiment primarily employs a semantic similarity matching strategy. This strategy involves constructing an exemption intent semantic core library, encompassing various semantic cores related to queries, waiting, and processing. Next, text vectorization is performed, using a pre-trained language model to convert customer service statements and exemption intent core expressions into high-dimensional semantic vectors. Then, the cosine similarity between the customer service statement vector and each exemption intent core vector is calculated, and the highest similarity value is taken as the matching score. For example, a similarity threshold of 0.75 can be set; when the matching score exceeds this threshold, the customer service statement is deemed to contain an exemption intent.

[0618] In practice, regular expression matching can be used to quickly identify statements with obvious exemption intent. Then, a semantic similarity matching strategy can be used to further filter the remaining customer service statements whose meanings meet the exemption criteria but whose expression is not covered by the regular expression pattern. This ensures the accuracy of the exemption intent identification results. For example, the following is an example of exemption intent detection:

[0619] Customer service replied: "Please be patient, I'll check the system."

[0620] Regular expressions might not be able to match this colloquial expression. In this case, the system calculates semantic similarity and finds that the vector similarity between the sentence and the core expression of the exemption intent, "Please wait a moment, let me check the system," reaches 0.82, exceeding the threshold of 0.75. Therefore, the system successfully determines that the sentence contains an exemption intent and activates the exemption period.

[0621] In one implementation, timing the silence of a customer's speech includes: calculating the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer statement after the customer's speech, and using the time difference as the silence time.

[0622] The silence time can be expressed by the following formula:

[0623] ,

[0624] in, The end time for the customer's speech. The start time of the first customer service representative's statement after the customer's statement, or the first customer's statement. Indicates the duration of silence.

[0625] like >T, where T represents the preset mute threshold. This means that the customer service representative has not responded to this customer message for a long time. At this time, a mute mark is added to this customer message. The mute mark indicates that the customer service representative is suspected of muteing this customer message once.

[0626] like This indicates that customer service responded promptly or that the customer spoke continuously.

[0627] This embodiment uses the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer's statement after the customer's speech as the silence time, which reasonably describes the time range in which silence events may occur in actual dialogue scenarios, thereby enabling accurate identification of silence events.

[0628] In one implementation, the exemption period has an exemption duration, which is determined based on the complexity of the customer's problem and / or the customer service representative's attributes; wherein, the complexity of the customer's problem includes: simple problems, routine problems, and complex problems; and the customer service representative's attributes include: customer service representative level and customer service representative's proficiency in the business area.

[0629] In practice, the exemption period can be automatically adjusted based on the complexity of the customer's question and / or the customer service representative's attributes. Taking home appliance customer service as an example, common customer question types include pre-sales product inquiries, price and model comparisons, after-sales installation appointment inquiries, fault reporting, and complaint feedback. If the customer service representative is a senior pre-sales representative with many years of experience, the exemption period can be reduced according to the preset adjustment strategy when a customer inquires about pre-sales issues; if the customer service representative is a new after-sales representative, the exemption period should be extended when a customer inquires about pre-sales issues. If the customer service representative is only making a simple inquiry, the exemption period should be reduced; if complex operations are required, a longer exemption period should be granted.

[0630] To adapt to the varying management granularity requirements of different business scenarios, this embodiment provides multiple configurable exemption duration setting strategies. These are illustrated below through three specific scenario examples.

[0631] Example scenario 1: Setting an exemption duration strategy based on the complexity of the customer's problem.

[0632] This exemption duration setting strategy is suitable for business scenarios that prioritize simple, stable, and easily manageable rules. This strategy sets a fixed exemption duration based solely on the estimated complexity of the customer's problem, without considering individual customer service differences. Please refer to Table 1, which schematically illustrates one exemption duration setting strategy.

[0633] Table 1

[0634]

[0635] Table 1 categorizes issues into simple queries, routine processing, and complex operations based on their complexity. Table 1 also shows typical scenario descriptions, exemption durations, and specific operations that customer service representatives need to perform for different issue types.

[0636] Taking the exemption time setting strategy shown in Table 1 as an example, if a customer asks, "I lost the remote control for my air conditioner, how much does it cost to replace it?", the system recognizes this as a simple query and automatically sets a 30-second exemption time. Customer service needs to check the price of the replacement part and reply within this time. The advantage of this exemption time setting strategy is that the rules are transparent and the computational overhead is minimal.

[0637] Scenario Example 2: Fixed time settings based on customer service attributes.

[0638] This exemption duration setting strategy is suitable for scenarios that emphasize personalized management and fairness, have a highly homogeneous business type, and exhibit minimal differences in issue complexity. The exemption duration is primarily set based on the customer service personnel's own attributes. Its advantage lies in its extremely simple rules, but it does not consider differences in issue complexity. Please refer to Table 2, which schematically illustrates one exemption duration setting strategy.

[0639] Table 2

[0640]

[0641] Table 2 shows that the adjustment factors for the exemption duration are related to the customer service level and the matching degree of the customer service's business area. When calculating the exemption duration, both the customer service level and the matching degree of the customer service's business area need to be considered. Table 2 shows the specific exemption duration for different customer service levels and different business area matching degrees, as well as the relevant explanations for setting the exemption duration.

[0642] The calculation formula for the exemption duration is as follows: Exemption duration = Exemption duration corresponding to customer service level + Adjustment time corresponding to business area matching.

[0643] Taking a junior customer service representative handling a cross-domain issue as an example, the exemption time for junior customer service representatives is 40 seconds, and the exemption time for business domain matching is 15 seconds. Therefore, the final exemption time is: 0 seconds + 15 seconds = 55 seconds.

[0644] Scenario Example 3: Weighted settings for comprehensive question types and customer service attributes.

[0645] This embodiment, based on distinguishing problem types, further considers the differences in customer service personnel capabilities to set a baseline exemption duration and adjustment coefficient, making the exemption duration more reasonable while adapting to business needs. Please refer to Table 3, which schematically illustrates an exemption duration setting strategy for this scenario.

[0646] Table 3

[0647]

[0648] Table 3 shows that the adjustment factors for the exemption duration are related to the problem type, customer service level, and the degree of matching between the customer service representative's business area and the exemption duration. When calculating the exemption duration, it is necessary to consider the problem type, customer service level, and the degree of matching between the customer service representative's business area and the exemption duration simultaneously. Table 3 shows the specific exemption duration for different problem types, different customer service levels, and different degrees of matching between business areas, as well as the relevant explanations for setting the exemption duration.

[0649] The calculation formula for the exemption duration is as follows: Exemption duration = Exemption duration corresponding to the problem type × Customer service level coefficient + Business area matching adjustment time.

[0650] Taking a senior customer service representative handling a complex cross-domain operational issue as an example, the exemption time corresponding to the issue type is 90 seconds, the senior customer service representative's coefficient is 0.9, and the business domain matching adjustment time is increased by 15 seconds, resulting in a final exemption time of 90 seconds. 0.9 + 15 seconds = 96 seconds.

[0651] By flexibly setting the exemption duration, it is possible to fully adapt to the exemption requirements of different business scenarios, thereby improving the accuracy of the overall silence detection method.

[0652] It should be noted that the exemption duration setting strategy in this embodiment can be set according to needs and is not limited to the three exemption duration setting strategies mentioned above.

[0653] In one implementation, an exemption status management mechanism can also be set up. Specifically, if a new customer service message is detected during the exemption period, the exemption period is terminated.

[0654] Specifically, the exemption status management mechanism also includes: if a new customer service message contains an intention to grant an exemption, a new exemption period will be reactivated starting from the end time of the new customer service message.

[0655] For example, such as Figure 3 As shown, the process of the exemption status management mechanism is as follows:

[0656] Step S301: When an exemption intent is detected, the exemption period is activated starting from the end of the current customer service speech.

[0657] Step S302: During the exemption period, no mute timer is set for new customer service messages.

[0658] Step S303: If the customer service representative responds to the customer within the exemption period, the exemption period ends immediately.

[0659] Step S304: If the exemption period expires and customer service still does not respond, the exemption status is terminated and it is immediately determined that a mute event has occurred.

[0660] Step S305: Output the judgment result of the silence detection.

[0661] For example, the following uses "Air Conditioner Repair Progress Inquiry" as an example to illustrate... Figure 3 The exemption status management mechanism is explained in detail. Sentences 3 to 5 in the "Air Conditioner Repair Progress Inquiry" section are as follows:

[0662] Sentence 3: The customer service representative said, "Let me check the progress of your work order; it will take a little time."

[0663] Sentence 4: The customer says, "Okay, also, my air conditioner is making strange noises."

[0664] Sentence 5: Customer service representative's message: "Found it, your work order is being processed. Regarding the abnormal noise issue you just mentioned, I need to understand it in more detail."

[0665] Sentence 3 is a customer service statement, so the first step is to check if sentence 3 contains an intention to grant an exemption. Since the keywords "check it out" and "it will take a little time" were detected, it can be determined that sentence 3 contains an intention to grant an exemption. Taking the above scenario example 3 as an example, combining the problem type (routine processing, 60 seconds) and the customer service level (intermediate, coefficient 1.0), and setting the exemption time to 60 seconds, the exemption period is represented as follows: , This indicates the end time of sentence 3, in seconds.

[0666] Next, regarding sentence 4, assuming that sentence 4 occurs within the exemption period, then silence detection will not be performed on sentence 4 during the exemption period.

[0667] Finally, regarding sentence 5, assuming that the customer service representative replied 45 seconds after the start of the exemption period, then the customer service representative's reply was within the exemption period, and the exemption status terminated.

[0668] This embodiment sets up a complete management mechanism of "activation → termination → reactivation" for the exemption period, which fully adapts to all possible situations during the exemption period, improves the flexibility and rationality of the exemption period mechanism, and thus improves the accuracy of the silence detection results.

[0669] In one embodiment, the above-mentioned silence detection method further includes: if the silence time obtained by the silence timing is greater than the silence threshold, marking the customer's speech with a silence mark; if the customer service representative still has not responded by the end of the exemption period, marking the customer service representative's speech corresponding to the exemption period with a silence mark. This embodiment achieves accurate positioning of different silence events by adding silence marks, which facilitates subsequent review of the silence detection results.

[0670] In one embodiment, the above-mentioned silence detection method further includes: taking a customer's speech with a silence mark as the target customer's speech, performing a dialogue flow logic check on the target customer's speech, and detecting whether the previous sentence of the target customer's speech was a customer service speech; if the previous sentence of the target customer's speech was a customer service speech, and the end time of the target customer's speech is earlier than the end time of the previous customer service speech, canceling the silence mark of the target customer's speech.

[0671] In practice, user interruptions may occur, leading to disordered sentence order in the structured text. This is because in actual conversations, the same speaker may pause naturally, or one party may interrupt the other. An ideal Automatic Speech Recognition (ASR) system should be able to segment sentences containing long pauses appropriately and accurately present interaction sequences such as "customer service - customer (interruption) - customer service". However, ASR models often suffer from the following two types of problems due to coarse or incorrect sentence segmentation granularity:

[0672] 1. Incorrectly merging spoken content from the same person, including natural pauses, into a single text;

[0673] 2. Cover up other people's interruptions within the same speaker's text block.

[0674] Both of these errors distort the original temporal logic of the dialogue, leading to misjudgments in silence detection.

[0675] Therefore, this embodiment proposes a dialogue flow logic verification mechanism. Its core principle lies in using the precise start and end timestamps of each sentence in the structured text to detect timing inconsistencies, thereby inferring potential merging or segmentation errors in ASR transcription. The specific verification rule is as follows: when a silence marker is detected for a customer's statement, if the preceding sentence is a customer service statement, and the end time of this customer's statement is earlier than the end time of the previous customer service statement (customer interruption), then it indicates an ASR transcription anomaly or customer interruption. In this case, the system will rule out the possibility of customer service silence and remove the silence marker from that customer's statement.

[0676] For example, as shown in Table 4, Table 4 presents a structured text of a conversation between a customer service representative and a customer to verify the delivery address.

[0677] Table 4

[0678]

[0679] In response to this dialogue, execute as follows Figure 4 The silence detection process shown, which includes a dialogue flow logic verification mechanism, includes the following steps:

[0680] Step S401, Initial silence detection.

[0681] For customer sentence 5 (content "201") in Table 4, calculate the interval between its end time and the start time of the next sentence (customer sentence 6): ΔT = t_start(sentence 6) - t_end(sentence 5) = 00:17:55 - 00:17:42 = 13 seconds.

[0682] Assuming the silence threshold T = 5 seconds, since 13 seconds > 5 seconds, the system initially flags the customer service representative for a single instance of silencing the customer in sentence 5.

[0683] Step S402: Trigger dialogue flow logic verification.

[0684] The system applies validation rules to check the contextual timing of the client's sentence 5:

[0685] Check if the sentence preceding customer sentence 5 (sentence 4) was spoken by customer service. The result is yes.

[0686] Check if the end time of customer sentence 5 (00:17:42) is earlier than the end time of the previous customer service statement (sentence 4) (00:17:52). Since 00:17:42 < 00:17:52, the result is yes.

[0687] Step S403, Analysis and Judgment.

[0688] The aforementioned temporal inconsistency (the customer finished speaking before the customer service representative finished speaking) reveals that the actual dialogue process should have been as follows: after the customer service representative repeated the address "Building 5," the customer immediately interrupted to confirm "201," and then the customer service representative continued with the subsequent remarks. The ASR system incorrectly merged the two segments of the customer service representative's speech before and after the interruption into sentence 4, thus concealing the fact of the interruption and causing sentence 5 to be misjudged as an independent, unanswered customer question.

[0689] Step S404: Perform the correction.

[0690] Based on the dialogue flow verification results, the system determined that this silence marker was caused by an ASR transcription anomaly (the interruption was not correctly segmented), and not by a customer service error. Therefore, the silence marker for customer sentence 5 was removed, and the silence detection result was corrected.

[0691] Through this mechanism, this embodiment can effectively identify and correct mute misjudgments caused by ASR sentence merging, interruption and segmentation errors, etc., and significantly improve the accuracy and reliability of mute detection in complex dialogue interaction scenarios.

[0692] In one implementation, the above-mentioned silence detection method also includes a short response recognition mechanism. This mechanism addresses silence situations caused by short customer service responses being missed during ASR transcription. Specifically, during actual conversations, while the customer is describing their situation, the customer service representative is often in a state of reassuring or listening to the customer, and frequently provides short responses. These short responses are easily missed during ASR transcription, and thus, during text-based detection, the customer service representative may be mistakenly judged as violating the silence rule.

[0693] To address this issue, a brief response recognition mechanism can be adopted, which includes: after performing dialogue flow logic verification, for the target customer's speech that still has a silence mark, detecting whether there are preset customer service voice features in the context audio segment corresponding to the target customer's speech in the customer service audio; if customer service voice features are detected in the context audio segment, canceling the silence mark of the target customer's speech, re-transing the customer service audio into structured text, and re-executing the silence detection method on the re-transcribed structured text.

[0694] In practice, call audio is handled in two ways. The first is dual-channel, which includes two independent audio streams: customer audio and customer service audio. The customer audio only contains all the sounds from the customer service side. The second is single-channel, which has only one mixed audio stream, where the customer service and customer voices are recorded together. The audio includes all the sounds from both sides.

[0695] For example, for dual-channel call audio, a short response recognition mechanism includes:

[0696] For target customer statements that still retain a silence flag after the dialogue flow logic verification, an audio segment within the first time interval is selected from the customer service audio as the context audio segment corresponding to the target customer statement. The presence of preset human voice features is then detected within this context audio segment. The first time interval is represented as [t1, t2 + T], where t1 represents the start time of the target customer service statement, t2 represents the end time of the target customer service statement, and T represents the silence threshold. If preset human voice features are detected in the context audio segment, the customer service audio is re-transcribed into structured text, and the aforementioned silence detection method is repeated on the re-transcribed structured text; otherwise, the silence flag for the target customer statement is maintained.

[0697] Specifically, the aforementioned voice features refer to customer service voice response features. Customer service voice responses typically contain specific short customer service phrases. Therefore, customer service voice response features can be constructed based on this characteristic to distinguish them from environmental noise features in call audio. For example, the specific short customer service phrase features shown in Table 5 can be used.

[0698] Table 5

[0699]

[0700]

[0701] Based on the customer service voice response features described in Table 5, five feature dimensions of the customer service speech are detected in the first time interval [t1, t2+T], including:

[0702] Duration detection: A short-time energy thresholding method is used. First, the short-time energy of audio frames within a first time interval is calculated. A noise floor threshold is set (e.g., -40dB), and continuous speech segments with frequencies above the noise floor threshold are extracted. The total duration of these continuous speech segments is then calculated. If it falls within the duration range of "customer service short phrase characteristics": 120ms≤ If the duration is ≤700ms, the duration detection result is recorded as "satisfied"; otherwise, it is recorded as "not satisfied".

[0703] Fundamental frequency detection: Fundamental frequency estimation is performed using the autocorrelation function method or the YIN algorithm. If a valid fundamental frequency is detected... It falls within the male (80-180Hz) or female (160-320Hz) range, and the fundamental frequency change rate within the segment is... If the frequency is less than the preset threshold, it indicates a constant frequency of non-mechanical noise, and the result is recorded as "satisfied" for the fundamental frequency detection; otherwise, it is recorded as "not satisfied".

[0704] Energy detection: The energy spectrum is calculated using short-time Fourier transform, and the energy envelope is extracted. The difference between the maximum energy Em within a segment and the background noise E is calculated. .like If the value is greater than 6dB and the variance σE of the energy envelope is greater than the preset threshold, it indicates fluctuations and non-stationary noise. In this case, the energy detection result is recorded as "satisfied"; otherwise, it is recorded as "unsatisfied".

[0705] Zero-crossing rate detection: Directly count the number of zero-crossing points in the audio frame and calculate the average value ZCRavg per unit time. If ZCRavg falls in the voiced area or transition area, or shows obvious segmented changes, such as high at the beginning and low at the end, and there are no constant extreme values ​​across the entire frequency band, such as the constant median of white noise, then the zero-crossing rate detection result is recorded as "satisfied". Otherwise, it is recorded as "not satisfied".

[0706] Formant detection: Linear predictive coding (LPC) analysis is used to extract the frequencies and bandwidths of the first 4-5 formants. If at least 2 significant formant peaks are detected, and their frequency positions conform to the typical distribution of human vowels or nasal sounds, such as F1 between 300-1000Hz, and the formant frequency changes smoothly over time, then the formant detection result is recorded as "satisfied"; otherwise, it is recorded as "not satisfied".

[0707] After the testing is completed, a comprehensive judgment is made, and the test results of the above 5 dimensions are weighted and scored. Either of the following two methods can be used:

[0708] (1) Full match mode: If the detection results of all 5 dimensions are marked as "satisfied", then [t1,t2+T] is directly determined as "suspected customer service response". At this time, the structured text transcription process needs to be re-executed, and the above-mentioned silence detection method is executed again on the re-transcribed structured text.

[0709] (2) Weighted threshold mode: Considering the complexity of the actual environment, a certain degree of fault tolerance is allowed. The system sets a weight for each dimension (e.g., fundamental frequency 0.3, formant 0.25, duration 0.20, energy 0.15, zero crossing rate 0.10). If the number of feature dimensions is ≥4, or the total weighted score is ≥0.8 (out of 1.0), then [t1, t2+T] is determined to be "suspected customer service response". At this time, the structured text transcription process needs to be repeated, and the above-mentioned silence detection method is executed again on the re-transcribed structured text.

[0710] If the result is "no customer service response", it means that the main interference during that period was environmental noise or non-human voice, and the customer service did not speak. This is considered "silent", and the structured text transcription reprocessing process will not be triggered.

[0711] It should be noted that the selection of the above-mentioned human voice features can be made adaptively according to the specific situation, and is not limited to the specific features shown in Table 5. This embodiment does not make specific limitations in this regard.

[0712] In one alternative approach, the short response recognition mechanism for single-channel call audio includes:

[0713] For target customer statements that still have a silence flag after performing dialogue flow logic verification, an audio segment within the first time interval is selected from the customer service audio as the context audio segment corresponding to the target customer statement, and the presence of preset human voice features is detected within the context audio segment. The first time interval is represented as [t1, t2 + T], where t1 represents the start time of the target customer statement, t2 represents the end time of the target customer statement, and T represents the silence threshold.

[0714] If the voice characteristics of a second person other than the customer are detected within the first time interval, the call audio is re-transcribed into structured text, and the above-mentioned silence detection method is performed again on the re-transcribed structured text; otherwise, the silence mark of the target customer's speech is maintained.

[0715] Specifically, the aforementioned voice characteristics can be derived from the voice interjection characteristics shown in Table 6.

[0716] Table 6

[0717]

[0718] Based on the characteristics of human voice insertion shown in Table 6, the presence of human voice insertion in mono audio is detected within the first time interval. The main feature dimensions detected are as follows:

[0719] Fundamental frequency presence detection: Using a multi-fundamental frequency tracking algorithm or harmonic product spectrum, the customer's primary fundamental frequency trajectory is first determined. Modeling is performed, and within the first time interval, a search is conducted to determine if any of the following exists: The second significant fundamental frequency peak, in addition to Calculate the minimum interval between two fundamental frequencies. If a second fundamental frequency is detected within the time window Furthermore, its frequency falls within the human voice range of 80-320Hz, and ΔF>20Hz, ensuring that it is not an overtone interference from the same sound source, and the duration of the second fundamental frequency is >100ms; at this time, it is determined that there are multiple human voices in the detection interval, and the result of the fundamental frequency detection is recorded as "satisfied", otherwise it is recorded as "not satisfied".

[0720] Formant Detection: Using cepstral analysis combined with a formant tracker, LPC analysis is performed on the audio frames to extract formant sequences F1, F2, and F3. These are compared to the center frequency distribution Pclient of the formants during normal speech to detect the presence of a new formant Pnew whose center frequency differs significantly from Pclient (e.g., a shift exceeding 100Hz), and whose energy is significantly higher than the background noise floor. If at least two independent formant groups are detected (Group A corresponding to the customer and Group B corresponding to the interjector), and the formants of Group B exhibit dynamic changes similar to human voice pronunciation over time, it is determined that multiple voices are present within the detection interval. In this case, the formant detection result is recorded as "satisfied"; otherwise, it is recorded as "unsatisfied".

[0721] For single-channel mixed audio, the judgment logic is more stringent because the channels cannot be physically separated. Only when the results of both the "fundamental frequency presence" and "formant detection" dimensions are "satisfied" will the first time interval be marked as "suspected customer service response" and the ASR reprocessing process be triggered; otherwise, it means that the main component during that time period is environmental noise or customer monologue, and it is judged as "silence is true".

[0722] Furthermore, within the second time interval [t2, t2+T], if the customer does not speak, it is only necessary to determine whether the customer service representative responds. At this time, the audio only contains the customer service representative's voice, and the aforementioned method for detecting human voice features corresponding to dual-channel audio can be used for detection.

[0723] Optionally, for sentences identified as "suspected customer service responses," the audio from the customer service channel during that period can be verified to confirm whether the customer service representative actually responded. Verification methods can include ASR transcription verification or manual verification, as long as verification is feasible. For example, ASR transcription verification is used here as an example for explanation, as follows:

[0724] Resubmit the audio from the customer service channel during the time period of the sentence described as a "suspected customer service response" to ASR for transcription. If reprocessing yields text, it indicates that customer service did respond, confirming that ASR missed transcribing it. Add the newly transcribed content to the conversation text record and remove the corresponding mute marker. If reprocessing still yields no text, mark it as "invalid audio, customer service did not provide a substantive response."

[0725] For example, a customer says, "This summer was the first time I turned on the air conditioner, but it didn't cool down for a long time," during which the customer service representative responds softly with "Hmm." Because this response is brief, it might be missed by the ASR (Automatic Responsive Response) engine. If it's dual-channel audio, the system analyzes the suspected silence period [t1, t2+T] from five dimensions: fundamental frequency, formants, duration, energy, and zero-crossing rate. If all these dimensions match the characteristics of a brief customer service response, it's marked as a "suspected customer service response," and ASR reprocessing is triggered. If it's single-channel audio, the system analyzes it from two dimensions: fundamental frequency presence and formant detection. If both dimensions match the characteristics of a human voice interruption, it's marked as a "suspected customer service response," and ASR reprocessing is triggered. The ASR engine successfully transcribes "Hmm," thus eliminating the suspicion of silence.

[0726] This embodiment improves the accuracy of silence detection results by performing short customer service response detection based on human voice characteristics on single-frequency and dual-frequency channel scenarios respectively.

[0727] Optionally, the above-mentioned silence detection method may also include: outputting the silence detection result.

[0728] The test results include the test conclusion, the sequence number of the silent sentence, and the reason for the silence, such as... Figure 5 As shown. Examples of attribute descriptions included in the detection results are as follows:

[0729] Compliance determination: If the entire call is not muted, the result is: Compliance.

[0730] Single violation determination: If no mute count is configured in the detection rules, a violation will be output if the entire call is muted once. Here, "prolonged silence without exemption intent" and "failure to respond within the time limit despite exemption" are both counted as one mute.

[0731] Prolonged silence without exemption intent. When a customer service representative fails to respond to a customer beyond the normal silence period without exemption intent, a violation will be marked on the "customer's message" where the representative did not respond. The silenced message number will be the customer's message, and the reason for silence will be "prolonged failure to respond to the customer."

[0732] Exemption granted but no response timed out. When a customer service representative intends to grant an exemption but fails to respond to the customer after the exemption inquiry period has expired, a violation will be marked on the sentence containing the exemption intention. The mute sentence number will be the sentence containing the exemption intention, and the violation reason will be "failure to respond to the customer after the exemption period has expired".

[0733] Cumulative count determination of violation: If the detection rule has a configured threshold N for the number of times the device has been muted (e.g., N=2), then:

[0734] When the number of times a call is muted exceeds the threshold N, the detection conclusion is output: violation, and the mute sentence number and mute reason of all mute events are output simultaneously.

[0735] When the total number of times the call is muted does not exceed the threshold N, the output conclusion is: compliant. However, the sequence number of the muted sentence and the reason for the muting event should still be output simultaneously.

[0736] The following provides a specific scenario for mute detection results. Assume the transcribed text structure of a call is as shown in Table 7, the system sets the standard mute threshold T=15 seconds, and the exemption time is uniformly 30 seconds.

[0737] Table 7

[0738]

[0739] The process for generating silence detection results is as follows:

[0740] (1) Analyze sentences 2 and 3.

[0741] Customer sentence 2 ends at 10:00:08. Customer service sentence 3 begins at 10:00:09. The waiting interval is 1 second, which is less than the standard mute threshold of 15 seconds, indicating that the customer service representative is not muted.

[0742] (2) Analyze sentence 3 and its following parts.

[0743] Sentence 3 contains the exemption intent keyword "Let me check for you." The system activates the exemption period, which begins at the end time of sentence 3 (10:00:11) and lasts for 30 seconds, ending at 10:00:41. During this period (10:00:11 to 10:00:41), customer messages (sentence 4) will not be muted. If customer service does not reply before 10:00:41, the exemption period will expire.

[0744] (3) Judgment.

[0745] Mark customer service sentence 3 as "Exemption granted but no response timed out". Record 1 instance of being muted. The mute sentence number is 3, and the reason for being mute is "no response to customer after the exemption time has expired".

[0746] (4) Analyze sentences 5 and 6.

[0747] Customer sentence 5 ends at 10:30:28. Customer service sentence 6 begins at 10:30:50. The waiting interval is 22 seconds, which is clearly greater than the usual threshold of 15 seconds. An examination revealed that the customer service representative did not speak between sentences 5 and 6 (no intention to waive this).

[0748] (5) Judgment.

[0749] Mark customer sentence 5 as "prolonged silence without exemption intent". Record 1 silence. The silence sentence number is 5, and the reason for silence is "no response to customer for a long time".

[0750] After obtaining the above silence test results, the following three different output formats can be used.

[0751] Scenario 1: The rule does not specify the number of times to mute (if it occurs, it is a violation).

[0752] If a call is marked as mute at least once, the conclusion is "violation". In this example, if two mute marks are marked, the output will list all mute events.

[0753] Output result: {

[0754] "Detection Conclusion": "Violation"

[0755] "Silence Incident": [

[0756] {

[0757] "Silent sentence number": 3,

[0758] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0759] },

[0760] {

[0761] "Silent sentence number": 5,

[0762] Reason for being silent: "Has not responded to customers for an extended period of time"

[0763] }

[0764] ]}.

[0765] Scenario 2: Rule configuration: Silence count threshold N=3 (cumulative violations).

[0766] A call is considered a "violation" if it contains at least three mute markers. In this example, two mute markers are present, which is compliant. The output should list all mute events.

[0767] {

[0768] "Test Conclusion": "Compliant"

[0769] "Silence Incident": [

[0770] {

[0771] "Silent sentence number": 3,

[0772] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0773] },

[0774] {

[0775] "Silent sentence number": 5,

[0776] Reason for being silent: "Has not responded to customers for an extended period of time"

[0777] }]}.

[0778] Scenario 3: Rule configuration: Silence threshold N=1 (cumulative violations).

[0779] The number of times the call was muted was 2, exceeding the threshold of 1, resulting in a "violation" conclusion, and a list of all muting events was provided.

[0780] Output result:

[0781] {

[0782] "Detection Conclusion": "Violation"

[0783] "Silence Incident": [

[0784] {

[0785] "Silent sentence number": 3,

[0786] Reason for being silent: "The customer has not been responded to after the exemption period has expired."

[0787] },

[0788] {

[0789] "Silent sentence number": 5,

[0790] Reason for being silent: "Has not responded to customers for an extended period of time"

[0791] }

[0792] ]}.

[0793] At this point, the silence detection result is output, and the silence detection is complete.

[0794] It should be noted that the above settings for violation rules in the output results can be adapted to meet specific needs, and this embodiment does not impose any specific restrictions on them.

[0795] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0796] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0797] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0798] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0799] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0800] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0801] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0802] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0803] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0804] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0805] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0806] 1) The technical solution of this application acquires call audio, performs structured text transcription on the call audio to obtain structured text, which includes customer service statements, customer statements, and timestamps corresponding to each statement; detects whether there is an exemption intention in the customer service statements indicating the need for additional processing time. If an exemption intention exists, an exemption period of a preset duration is activated starting from the end time of the customer service statement; for customer statements, if the start time of the customer statement is within the exemption period, the silence timer for the customer statement is skipped; if the start time of the customer statement is outside the exemption period, the silence timer for the customer statement is started; if the silence time obtained by the silence timer is greater than a preset silence threshold, or if the customer service has not responded by the end of the exemption period, a silence event is determined to exist. This application embodiment, through the exemption intention recognition mechanism, flexibly adjusts the silence detection threshold, effectively filters reasonable silence, and significantly improves the reliability of the detection results and the business adaptability.

[0807] 2) The technical solution of this application also introduces a dialogue flow logic verification mechanism, which detects and corrects word order abnormalities in structured text by comparing precise timestamps and eliminates false silence marks caused by interruptions, thereby significantly improving detection accuracy.

[0808] 3) The technical solution of this application also introduces a short response recognition mechanism to accurately identify short human voice responses on the customer service side, thereby achieving effective detection and correction of short responses.

[0809] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting silence, characterized in that, include: The call audio is acquired, and the call audio is processed by structured text transcription to obtain structured text, which includes customer service statements, customer statements, and timestamps corresponding to each statement. The system detects whether there is an intention to waive the requirement for additional processing time in the customer service statement. If such an intention exists, a waiver period of a preset duration is activated starting from the end time of the customer service statement. If the start time of the customer's speech is within the exemption period, the mute timer for the customer's speech is skipped. If the start time of the customer's speech is outside the exemption period, the customer's speech will be muted and timed. If the silence time obtained by the silence timer is greater than the preset silence threshold, or if customer service has not responded by the end of the exemption period, a silence event is confirmed.

2. The method according to claim 1, characterized in that, Detecting whether the customer service statement contains an intention to grant an exemption indicating the need for additional processing time, including: The customer service statement is subjected to regular expression matching. If a preset exemption statement is matched, it is determined that the customer service statement contains the exemption intention.

3. The method according to claim 1, characterized in that, Detecting whether the customer service statement contains an intention to grant an exemption indicating the need for additional processing time, including: Semantic recognition is performed on the customer service statement. If a preset exemption semantic is identified, it is determined that the customer service statement contains the exemption intention.

4. The method according to claim 1, characterized in that, The process of timing the silence during the customer's speech includes: Calculate the time difference between the end time of the customer's speech and the start time of the first customer service statement or the first customer's statement after the customer's speech, and use the time difference as the silence time.

5. The method according to claim 1, characterized in that, The exemption period has an exemption duration, which is determined based on the complexity of the customer's problem and / or the customer service representative's attributes. The complexity of customer issues includes: simple issues, routine issues, and complex issues; The customer service attributes include: customer service level and customer service proficiency in the business area.

6. The silence detection method according to claim 1, characterized in that, The method further includes: If a new customer service message is detected during the exemption period, the exemption period will be terminated.

7. The method according to claim 5, characterized in that, The method further includes: If the new customer service message contains the intention to waive the waiver, a new waiver period will be reactivated starting from the end time of the new customer service message.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If the silence time obtained by the silence timer is greater than the silence threshold, a silence mark is added to the customer's speech; If the customer service representative does not respond by the end of the exemption period, the customer service representative's message corresponding to the exemption period will be marked with the mute mark.

9. The method according to claim 8, characterized in that, The method further includes: The customer's speech with the mute tag is taken as the target customer's speech. The dialogue flow logic is checked on the target customer's speech to detect whether the previous sentence of the target customer's speech was a customer service speech. If the previous sentence of the target customer's statement was a customer service statement, and the end time of the target customer's statement is earlier than the end time of the previous customer service statement, then the mute mark of the target customer's statement is removed.

10. The method according to claim 9, characterized in that, The method further includes: After performing the dialogue flow logic verification, for the target customer who still has the silence mark speaking, detect whether there are preset customer service voice features in the context audio segment corresponding to the target customer's speech in the customer service audio; If the customer service voice feature is detected in the context audio segment, the mute marker of the target customer's speech is removed, the customer service audio is re-transcribed into structured text, and the mute detection method is executed again on the re-transcribed structured text.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 10.

12. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, said one or more programs being stored in the memory and configured to be executed by said one or more processors, said one or more programs including methods for performing any one of claims 1 to 10.

Images