An audio processing method and an auxiliary identification system for enhancing operation feedback in communication

Abstract

The application provides an audio processing method for enhancing operation feedback in communication and an auxiliary identification system, and relates to the fields of audio processing and communication technology.The audio processing method comprises steps of audio splitting, double-channel processing, AI identification and classification, mixing output and operation feedback, etc.The original audio stream is divided into a voice channel and a detection channel, the transient sound of the keyboard / mouse in the detection channel is enhanced, the operation sound type is identified by combining a CNN classifier, and corresponding operation feedback or identification prompts are output according to the application scene.The auxiliary identification system comprises multiple functional modules and supports client, server and hybrid architecture deployment.The application has the beneficial effects of enhancing operation sound feedback in e-commerce voice customer service and other scenarios, improving the user's perception of service professionalism, identifying suspected AI-assisted behavior in online interviews and filling the gap in the field of operation feedback enhancement and specific behavior identification in existing audio processing technology.

Description

Technical Field

[0001] This invention relates to the field of audio processing and communication technology, specifically to an audio processing method and auxiliary recognition system for enhancing operational feedback in communication. Background Technology

[0002] With the online and intelligent development of e-commerce services, voice customer service has become a crucial entry point for user inquiries, order guidance, and after-sales processing. During calls, customer service representatives typically need to simultaneously perform operations such as information retrieval, order verification, knowledge base searches, and form entry, generating keyboard typing and mouse clicks. In certain business scenarios, appropriately perceptible operational sounds can convey to users the signal that "in real-time query / processing" is underway, thereby enhancing users' perception of the service's professionalism and credibility.

[0003] On the other hand, with the popularization of generative AI capabilities, in scenarios such as online interviews and remote assessments, there are situations where test subjects use AI tools for real-time assistance during the call, often accompanied by frequent keyboard input and mouse copy-paste operations. Therefore, how to enhance and present these transient operation sounds in voice interaction, and how to recognize the "use of AI assistance" behavior when needed, has become a technical problem that needs to be solved.

[0004] Currently, mainstream voice call / conference systems generally integrate audio processing technologies such as intelligent noise reduction and echo cancellation. These technologies typically suppress or even eliminate background noise other than human voices, including keyboard sounds, mouse clicks, and fan noise. Existing "service quality monitoring" focuses more on voice dimensions such as speech rate, volume, keywords, and emotion; it lacks independent processing and presentation mechanisms for transient acoustic features such as keyboard / mouse input, tailored to business objectives. For scenarios requiring AI-assisted recognition, such as online interviews, existing solutions mostly rely on screen monitoring, camera behavior analysis, or log forensics, which suffer from high deployment costs and privacy compliance pressures.

[0005] Existing technologies suffer from the following problems: They weaken the perception of the service process, as existing noise reduction strategies suppress acoustic cues such as keyboard / mouse input for "real-time processing," making it difficult for users to perceive that customer service is querying or operating, thus reducing their sense of professionalism and trust. They lack controllable presentation methods; even if the business desires to retain some operational sounds, existing systems often lack the fine-grained control to "enhance only specific transient sounds without affecting the clarity of human voices." They struggle to support AI usage recognition; in scenarios such as online interviews, intensive keyboard / mouse operations may be strongly correlated with "using AI assistance," but existing audio links eliminate such sounds, making it impossible to provide real-time prompts or post-event evidence based on audio signals without adding additional sensors / monitoring methods. Summary of the Invention

[0006] 1. The technical problem that the invention aims to solve: This invention provides an audio processing method and auxiliary recognition system for enhancing operational feedback in communication. It can enhance operational sound feedback in service scenarios to improve user perception, and can also identify AI-assisted usage behavior in specific scenarios. It also features good privacy compliance and low resource consumption.

[0007] 2. Technical Solution: To achieve the above objectives, the present invention adopts a "dual-channel audio processing and feature enhancement" scheme. While retaining the human voice channel, a second channel is opened up to capture, separate and enhance the transient acoustic features of the mouse and keyboard, and output corresponding operation feedback or recognition prompts according to the application scenario.

[0008] An audio processing method for enhancing operational feedback in communication includes the following steps: Audio splitting: The original audio stream captured by the microphone during communication is copied into two channels, namely Channel A and Channel B. Channel A is the human voice stream, and Channel B is the detection stream. Channel A processing: Apply standard ANC noise reduction to channel A to filter out background noise and retain clear human voices; Channel B processing: Channel B undergoes high-pass filtering, transient feature extraction, and enhancement processing sequentially to preserve high-frequency transient signals and enhance their auditory quality. The cutoff frequency of the high-pass filter is set to 2kHz to remove most of the fundamental frequency of human voices and suppress interference. The transient feature extraction and enhancement processing includes: (a) Energy detection: Calculate the short-time energy of channel B audio and set the energy threshold Tenergy; (b) Starting point detection: Spectral flux is used to calculate signal abrupt change points to pinpoint the time when a keystroke or mouse click occurs; (c) Signal gain: Apply a dynamic gain of 6-12dB to the located transient segments to enhance the sharpness of the signal. AI recognition and classification: The enhanced channel B audio segment is converted into a Mel spectrogram, which is then input into a lightweight CNN classifier to identify the audio category as one of Silence, Speech, Keyboard, and Mouse_Click. In this technical solution, Silence, Speech, Keyboard, and Mouse_Click are respectively mute, human voice, keyboard sound, and mouse click sound. Mixing Output: The enhanced audio from channel B, identified as Keyboard or Mouse_Click, is superimposed onto channel A at a preset volume ratio to generate the final audio output; in scenarios requiring enhanced perception during e-commerce voice customer service, the volume ratio of the enhanced audio from channel B is set to 20% during mixing. Operation Feedback and Recognition Prompts: Real-time statistics of Keyboard or Mouse Click operation frequency per unit time, outputting corresponding operation feedback or recognition prompts according to the operation scenario; operation scenarios include service process awareness enhancement scenario and AI-assisted use recognition scenario; when in the service process awareness enhancement scenario, real-time operation feedback is perceived by the receiver through audio mixing; when in the AI-assisted use recognition scenario, if the operation frequency exceeds a preset threshold, a visual alarm prompt is triggered, the preset threshold being more than 20 keystrokes or mouse clicks within 5 seconds; The event timeline generation process involves the system automatically marking the time points corresponding to high-frequency operations after the communication ends, forming a traceable event timeline.

[0009] An auxiliary recognition system for enhancing operational feedback in communication includes an audio acquisition module, an audio splitting module, a human voice processing module, a transient feature processing module, an AI recognition and classification module, a mixing output module, an operational feedback module, and a storage module. Audio acquisition module: used to acquire the raw audio stream during communication; Audio splitting module: used to copy the original audio stream into channel A and channel B and output them to the voice processing module and transient feature processing module respectively; Voice processing module: Used to perform conventional ANC noise reduction on channel A to output clear human voice; Transient feature processing module: used to perform high-pass filtering, energy detection, start point detection and signal gain processing on channel B, and output the enhanced high-frequency transient signal; AI recognition and classification module: used to convert enhanced high-frequency transient signals into Mel spectrograms and identify audio categories using a lightweight CNN classifier; Mixing output module: Used to superimpose the enhanced signal identified as keyboard sound or mouse click sound onto clear human voice according to a preset ratio to generate the final audio; Operation feedback module: used to count the operation frequency based on audio category and output operation feedback or recognition prompts in combination with application scenario; Storage module: Used to store raw audio, processed audio data, and event timeline information.

[0010] Furthermore, the operation feedback module includes an operation activity dashboard, which displays the operation activity in real time. When the operation frequency exceeds a preset threshold, a red flashing warning is displayed along with a prompt message, suggesting that AI tools are being used.

[0011] Furthermore, the system can be deployed on the client, server media processing end, or adopt a hybrid architecture. In the hybrid architecture, the acquisition end performs audio preprocessing, and the cloud or playback end performs recognition and synthesis processing. The client includes the acquisition end or the playback end.

[0012] 3. Beneficial effects: Compared with the prior art, the technical solution provided by this invention has the following advantages: Enhance service perception: By selectively amplifying transient operation sounds such as keyboard / mouse input, users can clearly perceive the real-time processing of the service provider without affecting the clarity of human voices, significantly improving the professionalism and credibility of the service. Achieve accurate identification and prompts: In scenarios such as online interviews, by detecting and analyzing high-frequency operation sounds, it is possible to identify suspected AI-assisted behaviors in real time and issue prompts, providing a basis for subsequent verification; Excellent privacy compliance: It only analyzes and processes the audio stream during communication, without collecting sensitive information such as user screen images and biometrics, so users have little resistance and it complies with relevant privacy protection regulations; Low resource consumption: By using high-pass filtering and energy detection as pre-screening, deep learning inference is only performed on suspected transient segments, which greatly reduces the computational load on the client or server and supports real-time operation; Flexible application scenarios: It can be adapted to various communication scenarios such as e-commerce voice customer service, online interviews, and remote assessments. It also supports client-side, server-side, and hybrid architecture deployments, with strong compatibility. Dual feedback protection: Combining the dual feedback mechanism of "auditory enhancement" and "visual alarm" effectively reduces the false negative rate and improves the user experience.

[0013] It should be noted that the structures not described in this invention are not related to the design points and improvement directions of this invention, and are the same as or can be implemented using existing technologies, so they will not be elaborated here. Attached Figure Description

[0014] Figure 1 This is a system processing flowchart of the present invention; Figure 2 This is the logic diagram of the key algorithm of the present invention. Detailed Implementation

[0015] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.

[0016] Example 1: E-commerce voice customer service scenario System Deployment: Integrate the auxiliary recognition system of this invention into the server media processing end of the e-commerce voice customer service system; Audio Acquisition and Splitting: When customer service representatives talk to users, the audio acquisition module acquires the original audio stream from the customer service end, and the audio splitting module copies it into channel A and channel B. Channel processing: Channel A undergoes conventional ANC noise reduction via the voice processing module to filter out ambient noise and retain clear customer service voices; Channel B undergoes 2kHz high-pass filtering via the transient feature processing module to remove the fundamental frequency of the voice, and then uses energy detection and start point detection to locate transient signals of customer service keyboard typing and mouse clicking, and applies an 8dB dynamic gain enhancement signal. AI recognition and classification: The enhanced Channel B audio is converted into a Mel spectrogram, which is then input into a lightweight CNN classifier for recognition, confirming the customer service's keyboard input and mouse operation sounds; Mixing output: The intelligent mixer superimposes the enhanced operation sound of channel B onto the clear vocals of channel A at a volume ratio of 20%, generating the final audio that is transmitted to the user. Operational feedback: Users can clearly hear the customer service representative speaking through the headphones, and also perceive appropriate keyboard typing and mouse click sounds, indicating that the customer service representative is performing operations such as information inquiry and order verification in real time, thereby enhancing their recognition of the professionalism of the service; Data storage: The storage module records the audio data of the entire call and the time points corresponding to customer service operations, forming an event timeline, which facilitates subsequent service quality review.

[0017] Example 2: Online Interview Scenario System Deployment: A hybrid architecture is adopted, with the interviewee's client acting as the acquisition end for audio preprocessing, the cloud performing recognition and synthesis processing, and the interviewer's client integrating an operation feedback module; Audio Acquisition and Splitting: During the interview, the interviewee's client audio acquisition module acquires the raw audio stream, which is then copied into Channel A and Channel B by the audio splitting module. Channel A is preprocessed and then transmitted to the cloud to preserve the human voice, while Channel B is preprocessed and then transmitted to the cloud for transient feature processing. Feature extraction and recognition: The cloud-based transient feature processing module performs high-pass filtering, energy detection, start point detection, and 10dB gain processing on channel B, and then converts it into a Mel spectrogram. A CNN classifier is then used to identify whether the sound is a keyboard or mouse operation. Frequency statistics and judgment: The frequency counter counts the frequency of the interviewee's operation within a unit of time in real time. When 25 keystrokes are detected within 5 seconds, it is suspected that the interviewee is inputting a question into the AI ​​tool and is judged as exceeding the frequency limit. Feedback and Alarm: The intelligent mixer amplifies the operation sound by adding it to the interviewee's voice channel at 20% volume and transmits it to the interviewer's headphones, allowing the interviewer to clearly perceive the operation sound; at the same time, the operation feedback module triggers the operation activity dashboard on the interviewer's client, which flashes red and displays the prompt message "Suspected of using AI tools"; Post-interview verification: After the interview, the event timeline generated by the storage module automatically marks the specific time periods of high-frequency operations. Interviewers can review the interview during those time periods by combining the video recordings to verify whether the interviewee used AI-assisted cheating.

[0018] Example 3: Remote Evaluation Scenario System deployment: The system is integrated as a standalone plug-in into the client of the remote evaluation platform, including the evaluator's end and the evaluated end; Function enabled: When the evaluator activates the "listen-through" mode, the system enters the AI-assisted recognition work state; Audio processing and recognition: When the person being evaluated performs keyboard input or mouse operation during the evaluation process, their client collects the original audio and processes it in separate streams. Channel B is enhanced and then transmitted to the evaluator's end. The CNN classifier identifies the type of operation sound and counts the frequency. Real-time prompt: When the evaluated person is detected to be clicking the mouse frequently, and the number of clicks reaches 35 times within 10 seconds, it is suspected that the AI-generated answer is being copied and pasted. The activity dashboard on the evaluater's end will flash red, and a "Suspected use of AI assistance" prompt box will pop up. Event logging: The system automatically records the start time, end time, and frequency of the high-frequency operation, forming an event timeline stored locally. After the evaluation is completed, the timeline can be exported and compared with the evaluation video.

[0019] Example 4: Anti-cheat scenario in FPS games System Deployment: Integrate the auxiliary recognition system into the client-side audio processing module or server-side media analysis node of the FPS game, and link it with the game operation log module; Audio Acquisition and Distribution: During gameplay, the client audio acquisition module collects the player's device operation audio in real time, including mouse click sounds. The audio distribution module then copies this audio into Channel A (game voice stream) and Channel B (operation detection stream). Channel processing: Channel A is filtered by the voice processing module to remove ambient noise and retain the player's voice; Channel B is processed by the transient feature processing module to perform a 2kHz high-pass filter to suppress game background sound effects interference, detect and locate the transient signal of mouse click by the starting point, and apply a 10dB dynamic gain to enhance signal details; AI recognition and feature comparison: The enhanced channel B audio is converted into a Mel spectrogram and input into a lightweight CNN classifier. At the same time, the microsecond-level time interval features of mouse clicks are extracted: Machine macro script clicks: regular intervals (variance ≤ 0.05ms) and high consistency of acoustic features (spectral waveform overlap ≥ 90%); Manual clicks: random intervals (variance ≥ 0.2ms) and natural fluctuations in acoustic features; Anti-cheat detection: If the system identifies the characteristics of a machine macro script, it sends a "suspected cheat" flag to the game server and triggers an in-game pop-up message, such as "Abnormal operation behavior detected, please play the game properly"; Data retention: The storage module records audio clips, time interval data, and judgment results of abnormal operations, which serve as the basis for subsequent verification by the game platform.

[0020] Example 5: Smart Home / Security Scenarios System Deployment: Integrate the system into smart home audio acquisition terminals, such as smart speakers and security cameras with microphones, and link them with home security alarm systems; Audio Acquisition and Distribution: The terminal acquires indoor ambient audio in real time, and the audio distribution module copies it into Channel A (normal ambient sound) and Channel B (abnormal sound detection stream). Channel processing: Channel A maintains normal playback; Channel B undergoes a 1.5kHz high-pass filter through the transient feature processing module to highlight high-frequency transient sounds such as knocking and breaking. Suspected abnormal sounds, such as knocking, glass breaking, and metal impact of door locks, are located through energy detection and start point detection. A 12dB dynamic gain is applied to enhance signal identification. AI classification and recognition: The enhanced channel B audio is converted into a Mel spectrogram, input into a CNN classifier, and the audio category is identified as "normal household sounds", "abnormal knocking sounds", "glass breaking sounds", and "metallic door lock sounds". Security Trigger: If an abnormal category is identified, the system sends a trigger signal to the home security system, which will trigger the camera to record and push an alarm notification to the user's mobile phone, such as "A sound of breaking glass has been detected. Please pay attention to indoor safety." Linkage optimization: When scene conditions permit, the audio recognition results can be verified a second time by combining the image data of security cameras to reduce the false alarm rate.

[0021] Example 6: User Behavior Analysis System Deployment: Deploy the system on the terminal under test for software usability testing and interface with the test task management module; Audio Acquisition and Splitting: During the test, the terminal acquires the user's operation audio, and the audio splitting module copies it into Channel A (user voice feedback) and Channel B (operation sound detection stream). Channel processing: Channel A retains user voice feedback; Channel B undergoes 2kHz high-pass filtering via the transient feature processing module to enhance the transient sound of keyboard / mouse operations, and marks the time point of each operation through start point detection; Behavioral data statistics: When users complete a specific test task, the silent period of continuous inactivity (a single instance of ≥3 seconds is counted as a hesitation interval) is recorded, and the duration and time period of each hesitation interval are recorded; the number of operations per unit time is counted, such as the number of keyboard / mouse operations per minute, and the task phases with intensive operations are marked. Usability Analysis Report: After the test, the system combines the operation sound data and task completion time to generate a user behavior analysis report, presenting the "distribution of hesitation time at each stage of the task" and the "operation intensity curve" to help testers evaluate the usability of the software interface; Data application: The testing team optimized the software's interaction logic based on the report, improving the user experience.

[0022] In summary, as can be seen from the above specific embodiments, the present invention can realize operation feedback enhancement or AI-assisted use recognition functions according to the needs of different communication scenarios. While improving the service experience, it effectively solves the behavior recognition problem in specific scenarios and has advantages such as flexible deployment, privacy compliance, and low resource consumption, and has broad application prospects.

[0023] The above-described embodiments are merely illustrative of certain implementations of the present invention, and are described in a relatively specific and detailed manner. However, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements are all within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An audio processing method for enhancing operational feedback in communication, characterized in that: Includes the following steps: S1. Audio splitting: The original audio stream captured by the microphone during communication is copied into two channels, namely Channel A and Channel B; S2. Channel A Processing: Apply standard ANC noise reduction processing to channel A to filter out background noise and retain clear human voices; S3. Channel B Processing: Channel B is sequentially subjected to high-pass filtering, transient feature extraction and enhancement processing to retain high-frequency transient signals and enhance their listening experience; S4. AI Recognition and Classification: Convert the enhanced Channel B audio segment into a Mel spectrogram, input it into a lightweight CNN classifier, and identify the audio category as one of Silence, Speech, Keyboard, or Mouse_Click; S5. Mixing Output: Enhance the audio from channel B, which is identified as Keyboard or Mouse_Click, and add it to channel A at a preset volume ratio to generate the final audio output; S6. Operation Feedback and Recognition Prompts: Real-time statistics of Keyboard or Mouse_Click operation frequency per unit time, and output corresponding operation feedback or recognition prompts according to the operation scenario.

2. The audio processing method for enhanced communication operation feedback according to claim 1, characterized in that: In step S3, the cutoff frequency of the high-pass filter is set to 2kHz to remove most of the fundamental frequency of human voice and suppress the interference of human voice on high-frequency transient signals.

3. The audio processing method for enhanced communication operation feedback according to claim 1, characterized in that: The transient feature extraction and enhancement process in step S3 includes: (a) Energy detection: Calculate the short-time energy of channel B audio and set the energy threshold Tenergy; (b) Starting point detection: Spectral flux is used to calculate signal abrupt change points to pinpoint the time when a keystroke or mouse click occurs; (c) Signal gain: Apply a dynamic gain of 6-12dB to the located transient segments to enhance the sharpness of the signal.

4. The audio processing method for enhanced communication operation feedback according to claim 1, characterized in that: The operation scenarios in step S6 include service process perception enhancement scenario and AI-assisted usage recognition scenario; when in the service process perception enhancement scenario, the receiver perceives real-time operation feedback through audio mixing output; when in the AI-assisted usage recognition scenario, if the operation frequency exceeds a preset threshold, a visual alarm prompt is triggered.

5. The audio processing method for enhanced communication operation feedback according to claim 4, characterized in that: The AI-assisted recognition scenario has a preset threshold of more than 20 keystrokes or mouse clicks within 5 seconds.

6. The audio processing method for enhanced communication operation feedback according to claim 4, characterized in that: The service process perception enhancement scenario is an e-commerce voice customer service scenario, and the volume ratio of channel B enhanced audio is set to 20% during the mixed output.

7. The audio processing method for enhanced communication operation feedback according to claim 1, characterized in that: It also includes an event timeline generation step. After the communication ends, the system automatically marks the time points corresponding to high-frequency operations to form a traceable event timeline.

8. An auxiliary identification system for enhancing operational feedback in communication, characterized in that: It includes an audio acquisition module, an audio splitting module, a human voice processing module, a transient feature processing module, an AI recognition and classification module, a mixing output module, an operation feedback module, and a storage module; Audio acquisition module: used to acquire the raw audio stream during communication; Audio splitting module: used to copy the original audio stream into channel A and channel B and output them to the voice processing module and transient feature processing module respectively; Voice processing module: Used to perform conventional ANC noise reduction on channel A to output clear human voice; Transient feature processing module: used to perform high-pass filtering, energy detection, start point detection and signal gain processing on channel B, and output the enhanced high-frequency transient signal; AI recognition and classification module: used to convert enhanced high-frequency transient signals into Mel spectrograms and identify audio categories using a lightweight CNN classifier; Mixing output module: Used to superimpose the enhanced signal identified as keyboard sound or mouse click sound onto clear human voice according to a preset ratio to generate the final audio; Operation feedback module: used to count the operation frequency based on audio category and output operation feedback or recognition prompts in combination with application scenario; Storage module: Used to store raw audio, processed audio data, and event timeline information.

9. The auxiliary identification system for enhanced communication operation feedback according to claim 8, characterized in that: The operation feedback module includes an operation activity dashboard, which displays the operation activity in real time. When the operation frequency exceeds a preset threshold, a red flashing warning is displayed along with a prompt message, indicating that AI tools are being used.

10. The auxiliary identification system for enhanced communication operation feedback according to claim 8, characterized in that: The system can be deployed on the client, server media processing end, or in a hybrid architecture. In a hybrid architecture, the acquisition end performs audio preprocessing, while the cloud or playback end performs recognition and synthesis processing.

Images