Banking point-oriented multi-modal voice interaction method, device, equipment and medium

Abstract

The application relates to a bank outlet-oriented multi-modal voice interaction method, device and equipment and a storage medium. The method comprises the following steps: performing far-field audio acquisition and sound source positioning on a bank outlet environment to obtain a sound source positioning result; performing face detection and distance estimation on the outlet environment to obtain a face target; performing space-time alignment and verification on the sound source positioning result and the face target to determine a main speaker in the face target; extracting a lip movement feature of the main speaker to obtain a lip movement feature of the main speaker; fusing and denoising the lip movement feature and an acoustic feature of an audio signal obtained through far-field audio acquisition to obtain target audio; performing voice recognition on the target audio to obtain target text; and determining a to-be-executed business skill according to a semantic analysis result of the target text based on a preset financial term library and interacting with a target user. The application can accurately lock the main speaker and correctly understand the semantic of the main speaker in a noisy environment of a bank outlet.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to a multimodal voice interaction method, device, equipment and storage medium for bank branches. Background Technology

[0002] With the deepening of digital transformation in the banking industry, the deployment of digital tellers and intelligent voice interaction terminals in bank branches is becoming increasingly widespread. However, the actual service environment of bank branches places far higher demands on voice interaction technology than typical indoor scenarios. Branch lobbies experience high foot traffic and diverse background noise sources, including announcements, conversations among multiple customers, and the operation of self-service equipment. Traditional single-audio noise reduction technologies struggle to effectively filter these noises while preserving the target customer's voice, leading to a significant drop in speech recognition accuracy. Scenarios where multiple people speak simultaneously are extremely common in the lobby. Traditional audio pickup devices cannot accurately distinguish between the main speaker and background noise, easily misinterpreting conversations of others as interactive commands, or missing the target customer's voice due to strong competing noise sources. Furthermore, in existing voice interaction systems, audio acquisition, sound source localization, noise reduction, speech recognition, semantic understanding, and result output operate independently, lacking state sharing and information feedback between each stage. The results of sound source localization are not used to guide the direction of the noise reduction beam, and even if visual perception exists, it is only used for simple face detection and does not participate in audio processing. The recognition engine still receives mixed audio that has not been confirmed by the speaker. The entire system lacks a complete collaborative link from perception to cognition, causing any shortcoming in any stage to be directly transmitted to the final interaction result.

[0003] Therefore, how to enable voice interaction systems to accurately locate the speaker, filter out noise interference, and correctly understand customer semantics in the noisy environment of bank branches has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of the above, this application provides a multimodal voice interaction method, apparatus, device and storage medium for bank branches, the purpose of which is to solve the above-mentioned technical problems.

[0005] Firstly, this application provides a multimodal voice interaction method for bank branches, the method comprising: Far-field audio acquisition and sound source localization were performed on the environment of bank branches to obtain sound source localization results; Face detection and distance estimation operations are performed on the network environment to obtain face targets; The sound source localization result is spatiotemporally aligned and verified with the face target to determine the speaker in the face target; Lip movement features of the speaker are extracted; The target audio is obtained by fusing and denoising the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition. Perform speech recognition on the target audio to obtain the target text; Based on the semantic analysis results of the target text using a pre-set financial terminology database, the business skills to be executed and the interaction with the target user are determined.

[0006] Secondly, this application provides a multimodal voice interaction device for bank branches, which includes: Positioning module: used to perform far-field audio acquisition and sound source localization in the bank branch environment, and obtain sound source localization results; Detection module: used to perform face detection and distance estimation operations on the network environment to obtain face targets; Determination module: used to perform spatiotemporal alignment and verification operations between the sound source localization result and the face target to determine the speaker in the face target; Extraction module: used to extract lip movement features from the speaker's lips to obtain the speaker's lip movement features; Fusion module: used to fuse and reduce noise based on the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition to obtain the target audio; Recognition module: used to perform speech recognition on the target audio to obtain the target text; Interaction module: Used to determine the business skills to be executed and interact with the target user based on the semantic analysis results of the target text according to the preset financial terminology database.

[0007] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the multimodal voice interaction method for bank branches as described in any embodiment of the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal voice interaction method for bank branches as described in any embodiment of the first aspect.

[0009] The technical solutions provided in this application have the following advantages compared with the prior art: By integrating far-field audio acquisition and sound source localization, visual speaker locking, audiovisual fusion noise reduction, multilingual recognition, and financial semantic parsing into a collaborative chain, this system enables sound source localization results to guide beam pointing and visual search, visual speaker locking results to control the selection of the pickup target, lip movement information to assist in audio noise reduction, and a financial terminology database to correct semantic parsing. Each step is seamlessly connected and shares state, solving the problem of poor interactive experience caused by the independent operation of each step in existing systems. This application can accurately locate the speaker in noisy environments and filter out background noise and interference from others, ensuring accurate acquisition and correct response of the digital human to customer speech. Simultaneously, lip movement features are reused for endpoint detection, making the recognition boundary more accurate. The enhancement of hot words in the financial terminology database improves the accuracy of professional terminology recognition, enabling accurate speaker location, noise filtering, and correct understanding of the speaker's semantics in noisy environments such as bank branches, thereby ensuring the correctness of business processing and customer satisfaction. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a preferred embodiment of the multimodal voice interaction method for bank branches according to this application; Figure 2 This is a schematic diagram of a preferred embodiment of the multimodal voice interaction device for bank branches according to this application; Figure 3 This is a schematic diagram of a preferred embodiment of the electronic device of this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0014] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0015] Reference Figure 1 The diagram shown is a flowchart illustrating an embodiment of the multimodal voice interaction method for bank branches according to this application. The method is executed by an electronic device, which can be implemented by a software system and / or a hardware system. This multimodal voice interaction method for bank branches includes: Step S10: Perform far-field audio acquisition and sound source localization on the bank branch environment to obtain the sound source localization results; Step S20: Perform face detection and distance estimation operations on the network environment to obtain face targets; Step S30: Perform spatiotemporal alignment and verification operations between the sound source localization result and the face target to determine the speaker in the face target; Step S40: Extract lip movement features from the speaker's lips to obtain the speaker's lip movement features; Step S50: Perform fusion and noise reduction based on the lip movement characteristics and the acoustic characteristics of the audio signal obtained from the far-field audio acquisition to obtain the target audio; Step S60: Perform speech recognition on the target audio to obtain the target text; Step S70: Based on the semantic parsing results of the target text using a preset financial terminology database, determine the business skills to be executed and interact with the target user.

[0016] This application provides a multimodal voice interaction method for bank branches. The method is executed by a multimodal voice interaction terminal deployed in the bank branch. The terminal integrates an 8-microphone linear microphone array, a 5-megapixel binocular camera, a speaker, an RK3588 octa-core processor and a 6TOPS neural network processor (NPU), an RS232 serial port and an HPOUT audio interface, and the operating system supports Android 12 or Android 13.

[0017] In the lobby of a bank branch, there are multiple sound sources, including announcements calling numbers, conversations among multiple customers, and the operation of mechanical equipment such as banknote counting machines. The terminal needs to use spatial filtering to separate the directional information of the effective sound sources from the mixed environment, so as to provide a priori location for subsequent speaker identification.

[0018] Once the terminal is powered on, it continuously collects far-field audio from the network environment using an 8-microphone linear array. The array employs a parallel linear topology, with an element spacing of 15 mm to 20 mm. The spacing between the seven elements forms an effective array aperture of approximately 105 mm to 140 mm. The array module size is approximately 35 mm × 8 mm, the element signal-to-noise ratio is greater than 63 dB, and the operating temperature range covers -20 degrees Celsius to 80 degrees Celsius. It can effectively capture human voice signals within a range of 3 to 5 meters.

[0019] The array converts the acquired sound into multiple digital audio signals, buffering and preprocessing them in 20-millisecond frames. After acquiring the multiple audio signals, the terminal calculates the cross-power spectrum (CPS) for any two array elements and performs PHAT-weighted whitening on the CPS, normalizing it using the amplitude of the CPS while retaining only phase information. The terminal then performs an inverse Fourier transform on the weighted CPS to obtain the time-domain cross-correlation function (TFC). The peak position of the TFC is detected, and the time shift corresponding to this peak position is the estimated time difference between the two signals. PHAT weighting ensures that the TFC exhibits a sharp peak at the true time delay, suppressing reverberation and noise interference on the amplitude of the TFC, thus maintaining robustness in the time delay estimation even under dotted reverberation conditions.

[0020] The terminal repeats the above time delay estimation for each array element combination, obtaining multiple sets of estimated time difference of arrival values. It then traverses each candidate direction within a 360° horizontal range with a step size of 1° to 3°. For each candidate direction, the terminal calculates the theoretical propagation time delay of each array element based on its geometric position and the candidate direction. The received signals from each element are aligned and superimposed according to the theoretical time delay, and the PHAT-weighted spatial spectral power is calculated. The terminal uses the candidate direction corresponding to the maximum spatial spectral power as the initial sound source direction estimate. Because the 8-microphone linear array itself has front-to-back mirror ambiguity—meaning sound sources located 30° in front of and 150° behind the array may produce the same time delay pattern on the linear array—the terminal uses the face orientation detected by binocular vision in subsequent steps to disambiguate this mirror direction, thereby achieving 360° omnidirectional unambiguous positioning.

[0021] After obtaining the initial sound source direction estimate, the terminal refines the spatial spectrum peak position using parabolic interpolation. This involves fitting the power values ​​of the peak point and its two adjacent sampling points to a parabola, and taking the vertex of the parabola as the precise peak direction, thus overcoming the limitation of angle step size on positioning accuracy. The terminal also performs median filtering on the sound source localization results of multiple consecutive frames to suppress positioning jumps caused by instantaneous noise. Ultimately, in a typical grid environment with a signal-to-noise ratio greater than 0 dB, the positioning error is stabilized within ±12°.

[0022] The terminal simultaneously runs a keyword wake-up detection mechanism, extracting Mel-frequency cepstral coefficient features from the single-channel audio after beamforming. A lightweight keyword detection network running on the processor's small core calculates the confidence score. When the confidence score exceeds the initial threshold, a more precise verification network is activated for further review. Only after both levels of detection pass is the wake-up considered valid. After wake-up, the terminal enters an interactive state. Accurate speech recognition distance is less than 5 meters; a recommended interaction distance is 0.8 to 1.2 meters. This sound source localization result provides acoustic direction for subsequent audio-visual spatiotemporal alignment.

[0023] Since single acoustic positioning can only provide the direction of the sound source, it cannot confirm whether the sound source comes from a customer with the intention to interact. The terminal needs to use visual information to verify the identity of the customer based on the acoustic positioning results, thereby accurately filtering out the customer currently interacting with the terminal from a multi-customer scenario. The terminal uses a 5-megapixel dual-lens camera to continuously capture video streams of the branch scene at a frame rate of 30 frames per second. The effective pixel of each lens is 1920×1080, the effective focal length of the lens is 2.88 mm, the aperture is F2.7, and the field of view is 102° diagonally, 89° horizontally, and 50° vertically, with an imaging distance covering from 70 cm to infinity. Calculated based on a horizontal field of view of 89°, the horizontal coverage width is approximately 2.0 meters at a typical interaction distance of 1.0 meter and approximately 6.0 meters at 3 meters, which can completely cover the waiting area and interaction area in front of the branch counter. The vertical field of view of 50° can cover the facial area of ​​customers of different heights. The binocular camera module integrates a 650 ± 10 nm infrared filter and supports automatic gain, automatic exposure, and automatic white balance, adapting to changes in dot lighting brightness. The terminal runs a lightweight anchor frame detection network deployed on the NPU for each frame of image. After INT8 quantization, this network achieves a single-frame inference time of less than 10 milliseconds on a 6 TOPS NPU, meeting the real-time processing requirement of 30 frames per second.

[0024] The detection network outputs the bounding box, confidence score, and facial landmarks of each face in the image. Facial landmarks include the positions of feature points such as the eyes, nose tip, and corners of the mouth. The terminal calculates the image azimuth angle of each face based on pixel coordinates and camera intrinsic parameters, and determines the facial orientation angle of each face based on the facial landmarks. Only faces with a facial orientation angle less than a preset threshold (i.e., faces facing the terminal) are given interaction candidate status, thus initially confirming the interaction intention. For each face with interaction candidate status, the terminal performs local stereo matching on its left and right eye images with the face detection bounding box as the constraint region, calculates the disparity value of the target face region, and then calculates the distance from each face to the terminal using the binocular ranging formula Z=B·f / D, where B is the binocular baseline distance and f is the equivalent focal length in pixels.

[0025] The terminal performs temporal Kalman filtering to smooth the ranging results of each frame, outputting a stable distance estimate. After obtaining the image position, azimuth, and distance information of each face target, the terminal performs spatiotemporal alignment of these visually detected face targets with the sound source localization results. For spatial alignment, since the camera and microphone array are rigidly mounted within the same module, the terminal pre-calibrates the rotation and translation extrinsic parameters between them. Based on this, the azimuth of each face target in the image pixel coordinate system is mapped to the microphone array coordinate system, ensuring that the acoustic direction and visual orientation are on the same spatial reference. For temporal alignment, audio frames and video frames are aligned with a unified timestamp. The audio frame length is 20 milliseconds, and the video frame interval is approximately 33 milliseconds, with the time error controlled within half a video frame period. The terminal interpolates the sound source localization results on the time axis to align with the sampling time of each face target's video frame, making the acoustic and visual data comparable in the temporal dimension. After spatiotemporal alignment is completed, the terminal compares the direction angle of the sound source localization result with the azimuth angle of each facial target. Simultaneously, the distance from each facial target to the terminal is used as one of the conditions for determining a valid interactive target. Visual targets whose direction matches the sound source localization result, face the terminal, and are within the effective interaction distance range are identified as the same target corresponding to the sound source, and this target is determined to be the speaker. The terminal writes the speaker's facial tracking bounding box, azimuth angle, and distance information into the speaker's register, and all subsequent processing steps use this speaker as the service target.

[0026] After identifying the speaker, the terminal needs to acquire dynamic information related to the speaker's vocalization from a visual perspective. This visual information is used in subsequent fusion and noise reduction to indicate the duration and intensity of the target speaker's voice, thus compensating for the deficiency of pure acoustic noise reduction in distinguishing the target speaker's voice from others' voices in non-steady-state noise environments. Using the lip keypoints among the speaker's facial keypoints as a benchmark, the terminal continuously extracts temporal images of the speaker's lip region of interest from the video stream captured by the binocular camera, forming a lip image sequence. The terminal performs scale normalization and grayscale processing on each frame of this image sequence to ensure consistency in the size and brightness of the lip region images. The terminal performs temporal analysis on the processed lip region image sequence, extracting dynamic change information that characterizes the degree of lip opening and closing, the amplitude of lip shape changes, and the rhythm of movement. This dynamic change information is encoded into a feature vector form, serving as the speaker's lip movement feature output. This lip movement feature reflects the speaker's lip movement state at each moment, allowing for determination of whether the speaker is vocalizing, providing a visual prior to the speech activity for subsequent fusion and noise reduction.

[0027] In bank branch environments, the terminal's own announcements, background steady-state noise, and sudden voices from others can all contaminate the audio signal. Pure acoustic noise reduction alone struggles to balance noise reduction depth and voice fidelity when the target voice and noise frequencies overlap, easily leading to mis-suppressing of the target voice or incomplete suppression of non-steady-state voices from others. To address this, visual lip movement information is incorporated into the noise reduction process. Lip movement characteristics are used to indicate the time-frequency regions where the target voice exists, enabling the noise reduction process to distinguish which signal components belong to the speaker and which are interference signals.

[0028] The terminal starts with multiple audio signals acquired from far-field audio acquisition. First, it performs spatial filtering on these signals in the acoustic domain. Using the sound source localization results, it guides the pickup beam towards the speaker's direction, enhancing the signal component from the target direction and suppressing interference from other directions. This spatial filtering employs a two-stage scheme combining fixed and adaptive beamforming. The first stage uses delayed summation fixed beamforming to point the main lobe towards the direction corresponding to the sound source localization results, initially enhancing the target speech and suppressing noise from other directions. The second stage, based on the fixed beam output, calculates the optimal weighting vector using the minimum variance distortion-free response criterion. This minimizes the array output power while ensuring distortion-free signal in the target direction, adaptively creating nulls in the interference direction and outputting the spatially filtered audio. The terminal then performs statistical spectrum estimation noise reduction on this spatially filtered audio, suppressing steady-state noise and slowly varying interference in the environment, and outputting the audio after acoustic domain noise reduction. This acoustic domain noise reduction constitutes the first layer of noise reduction.

[0029] The terminal then enters a second stage of noise reduction, combining visual and acoustic elements. It extracts acoustic spectral features from the acoustically denoised audio and simultaneously inputs lip movement features as visual prior information into the fusion noise reduction module. This module, based on the speaker's vocal timing and intensity indicated by the lip movement features, protects components of the acoustic spectrum belonging to the target human voice and suppresses components belonging to non-target human voices or pure noise. Guided by lip movement information, it achieves refined noise reduction and outputs the target audio. During this fusion process, the terminal also employs a protection strategy to avoid vocal distortion. The time-frequency masking coefficient is clamped above a preset lower limit to prevent excessive suppression of the target speech segment, which could lead to distortion. When the lip movement features determine that the speaker is speaking, gain compensation is applied to the masking coefficient for the corresponding time period to ensure the complete preservation of the target human voice. When the terminal detects that the confidence level of the visual branch is below a preset threshold, it automatically reverts to pure acoustic noise reduction mode and appropriately reduces the noise reduction intensity to avoid introducing erroneous visual information into the noise reduction process.

[0030] After dual noise reduction, the target audio has been filtered out of background noise and interference from others. However, the audio signal itself is still in waveform or spectral form. The terminal needs to convert it into text data through a speech recognition engine for subsequent semantic parsing module to understand the business logic. The terminal pushes the target audio to the input of the speech recognition engine in a circular buffer with a buffer depth of 200 milliseconds to absorb jitter and avoid accumulated delay. The engine receives the audio data stream to be recognized. During the recognition process, the terminal performs acoustic speech activity detection based on the spectral energy of the target audio and the modeled speech probability to determine the start and end boundary candidates of the speech segment. At the same time, the terminal reuses the speaker's lip movement features in the endpoint determination process, using the speaker's vocal state indicated by the lip movement features to verify the acoustically detected endpoint candidates.

[0031] When the acoustically detected speech start and end times match the phonation start and end times indicated by lip movement features, the terminal confirms the endpoint as valid. When they do not match, the terminal corrects the acoustic endpoint based on visual lip movement information, thereby avoiding sentence truncation or prolonged tail sounds due to acoustic misjudgment in noisy environments. After obtaining a reliable sentence endpoint, the terminal decodes the audio data to be recognized using this endpoint as the sentence boundary. It outputs the posterior probability of phonemes through a streaming acoustic model, and then performs a beam search on a weighted finite state converter graph through a decoder. Combining the comprehensive score from the language model, the terminal outputs the optimal word sequence as the target text.

[0032] Although the target text presents the speaker's voice content in text form, it is an unstructured natural language expression and may contain homophone errors or non-standard colloquialisms. Directly executing business based on this text is prone to errors. The terminal needs to convert it into structured business instructions and match them with corresponding service skills.

[0033] The terminal pre-builds and stores a financial terminology database for banking operations. This database covers business lines such as deposits, wealth management, loans, credit cards, account services, fund transfers, precious metals, and foreign exchange. It includes standard terminology entries, synonymous colloquial expressions, and business skill identifiers associated with each terminology entry. This database is stored locally on the terminal as a hot-updateable structured file and can be configured and expanded via a cloud platform. During speech recognition decoding, the terminal loads this terminology database and applies a probabilistic bias to language model paths containing these terms during the decoding search. This prioritizes the use of homophones and near-homophones for specialized terms such as structured deposits, large-denomination certificates of deposit, and net asset value-based wealth management, while simultaneously normalizing colloquial expressions to standard terminology, reducing ambiguity in subsequent semantic parsing stages.

[0034] After obtaining the target text, the terminal performs intent recognition and slot extraction, outputting a structured semantic result containing intent tags and slot fields. The intent tags represent the type of customer's business need, and the slot fields contain the specific business parameters involved in that need. The terminal then compares the intent tags and slot fields in the structured semantic result with the built-in skill matching criteria, calculating the matching score for each business skill. The business skill with the highest matching score exceeding a preset execution threshold is identified as the skill to be executed. When the matching score falls within the follow-up question range, the terminal generates a clarifying follow-up question to guide the customer to provide additional information. When no matching skill is available, the terminal switches to a fallback script or human customer service guidance.

[0035] The terminal ultimately interacts with the target user using this business skill. This includes displaying business guidance information on the terminal screen, broadcasting voice instructions through the speaker, and transmitting the target text, structured semantic results, and wake-up status to the digital human's main controller via an RS232 serial port using a unified frame protocol. The frame protocol includes a frame header, frame length, message type, sequence number, data payload, and CRC checksum fields. The receiver verifies the frame header, frame length, and CRC checksum and uses the sequence number to detect lost frames and request retransmissions. The terminal also outputs noise-reduced original audio through the HPOUT audio interface, allowing the digital human model to perform secondary processing on the precisely controlled audio, or enabling human agents to remotely monitor the customer's original voice for service quality checks.

[0036] By employing a front-end processing chain that integrates far-field audio acquisition and sound source localization, speaker identification, and visual lip movement and acoustic signal fusion for noise reduction, the system addresses the challenge of accurately capturing effective human voices in the complex acoustic environment of bank branches. An 8-microphone linear array, combined with time-of-arrival estimation and a controllable response power phase transformation algorithm, enables 360° sound source localization. The array element signal-to-noise ratio exceeds 63 dB, and the localization error remains stable within ±12° in typical branch environments, providing a reliable azimuth prior for beam pointing and avoiding wasted computational power due to indiscriminate sound pickup. A binocular camera performs face detection, facial orientation determination, and binocular parallax ranging. The detected facial targets are spatiotemporally aligned and cross-verified with the sound source localization results, achieving a face detection accuracy of 99.99%. This allows the system to accurately identify the speaker, eliminating false triggers and missed identifications caused by conversations with others. Simultaneously, a facial orientation filtering mechanism ensures that only customers with the intention to interact can trigger a system response. Based on this, the dual noise reduction stage uses lip movement characteristics to mark the time-frequency existence region of the target human voice to guide the noise reduction process. The first stage of acoustic noise reduction, combined with the second stage of audiovisual fusion noise reduction, achieves an echo cancellation of -25 dB and a front-end noise reduction of -10 dB. The protective gain compensation of the visual branch for the target human voice component ensures both noise reduction depth and human voice fidelity, solving the problems of human voice distortion and incomplete noise reduction when the spectrum overlaps in pure acoustic noise reduction.

[0037] The target audio, after dual noise reduction, is pushed to the speech recognition engine for recognition of Chinese, English, and dialects such as Cantonese, Minnan, and Sichuanese. Accurate recognition distance is less than 5 meters, with a recommended interaction distance of 0.8 to 1.2 meters. Lip movement features are reused for endpoint detection, allowing visual speech activity detection to verify acoustic endpoint candidates, avoiding sentence truncation or prolonged ending sounds due to endpoint misjudgment in high-noise environments. A financial business terminology database performs probability biasing and intent recognition slot extraction on the target text, prioritizing professional terms such as structured deposits, large-denomination certificates of deposit, and net asset value-based wealth management in homophone / near-phone competition. Combined with multiple professional skills and custom skill matching, this ensures accurate understanding and correct response to customer business needs. Ultimately, the target text, semantic results, and wake-up status are transmitted via RS232 serial port using a unified frame protocol with CRC check, and the original audio after noise reduction is output through the HPOUT audio interface. This forms a complete collaborative process from far-field acquisition, speaker locking, dual noise reduction, multilingual recognition to semantic parsing. Each step is seamlessly connected and shares status, enabling the digital human to accurately and efficiently serve every customer in the high-noise working environment of the bank, effectively improving business processing efficiency and customer service experience, and reducing operating costs.

[0038] In one embodiment, the step of performing spatiotemporal alignment and verification between the sound source localization result and the facial target to determine the speaker in the facial target includes: The sound source localization results are interpolated on the time axis to align with the sampling time of the video frames of each face target, and the azimuth angle of each face target in the image pixel coordinate system is mapped to the microphone array coordinate system to obtain the aligned face target and azimuth information. The corresponding orientation deviation tolerance is calculated based on the distance from each face target to the terminal, where the terminal is a multimodal voice interaction terminal in a bank branch. Calculate the original deviation between the sound source localization result and the orientation angle of each face target, and select face targets whose original deviation is less than the corresponding orientation deviation tolerance and whose faces are facing the terminal as candidate targets. If there is only one candidate target, then that candidate target will be the speaker. If there are multiple candidate targets, the one with the smallest distance to the terminal among all candidate targets will be selected as the speaker.

[0039] After obtaining the sound source localization results and detecting each facial target and its distance from the terminal, the terminal needs to perform audio-visual spatiotemporal alignment and cross-verification to accurately locate the speaker currently interacting with the terminal in a multi-person scenario. Since single acoustic localization can only provide the direction of the sound source, it cannot confirm whether the sound source comes from a customer with the intention to interact. However, there are often multiple sound sources such as conversations and broadcasts in the lobby of a branch. The terminal needs to use visual information to verify the identity of the acoustic localization results, thereby accurately filtering out the customer currently interacting with the terminal from a multi-customer scenario. Since the camera and microphone array are rigidly installed in the same module, the terminal obtains the rotation and translation extrinsic parameters between the two through pre-calibration. Based on this, the azimuth angle of each facial target in the image pixel coordinate system is mapped to the microphone array coordinate system, so that the acoustic direction and visual orientation are under the same spatial reference. Meanwhile, audio frames and video frames are aligned with a unified timestamp. The audio frame length is 20 milliseconds, and the video frame interval is about 33 milliseconds. The terminal interpolates the sound source localization results on the time axis to align with the sampling time of the video frames of each face target, so that the acoustic data and visual data are comparable in the time dimension. After the above spatial mapping and time interpolation, the terminal obtains the aligned face targets and their orientation information in the microphone array coordinate system.

[0040] After completing spatiotemporal alignment, the terminal determines the speaker for each face target that qualifies for interaction. Since the sound source localization error increases with target distance, a fixed tolerance would lead to mismatches for near-range targets due to an excessively wide tolerance, or missed detections for far-range targets due to an excessively narrow tolerance. Therefore, the terminal obtains the distance from each face target to the terminal and calculates a corresponding azimuth deviation tolerance for each face target based on this distance. A smaller azimuth deviation tolerance is set for near-range targets to tighten the screening criteria, while a larger azimuth deviation tolerance is set for far-range targets to avoid missed detections, thus achieving distance-adaptive tolerance configuration. The terminal then calculates the original deviation between the azimuth angle of the sound source localization result and the azimuth angle of each face target, compares this original deviation with the corresponding azimuth deviation tolerance of the face target, and determines whether the face of each face target is facing the terminal. Face targets with an original deviation less than the corresponding tolerance and whose face is facing the terminal are selected as candidate targets.

[0041] The terminal assesses the number of candidate targets. If there is only one candidate target, it directly identifies that target as the speaker. If there are multiple candidate targets (i.e., multiple customers are near the sound source and facing the terminal), the terminal identifies the target with the shortest distance to the terminal as the speaker. This distance-first arbitration rule aligns with the actual service scenario in bank branches, as the customer closest to the terminal is usually the one directly handling the current transaction. Once the speaker is identified, the terminal writes the speaker's facial tracking frame, azimuth, and distance information into the speaker register. All subsequent processing steps then use this speaker as the service target, ignoring other customers' voices. Through audiovisual spatiotemporal alignment and distance adaptive tolerance verification, the terminal can accurately locate the target customer currently interacting with it even in multi-customer, multi-sound-source scenarios, effectively preventing false triggers and missed identifications caused by others talking. Furthermore, the facial orientation screening and distance arbitration mechanisms ensure that only customers with the willingness to interact and within the effective interaction area can trigger a system response, improving the accuracy and user experience of the digital human service in bank branches.

[0042] In one embodiment, extracting lip movement features from the speaker's lips to obtain the speaker's lip movement features includes: Based on the key facial points of the speaker, the region of interest (ROI) image of the lips is cropped from the speaker's facial image. The ROI image of the lips is then normalized and grayscaled to obtain a normalized lip region image sequence. Based on the position of the lip key points among the facial key points, the speaker's lip opening degree, lip width, and lip corner displacement are calculated to obtain the speaker's lip geometric feature vector; The normalized lip region image sequence is input into a lightweight convolutional network for temporal feature encoding to obtain the lip appearance feature vector of the speaker. The lip geometric feature vector and the lip apparent feature vector are aligned temporally and then concatenated to obtain the speaker's lip movement features.

[0043] After locking onto the speaker, the terminal needs to extract lip movement features from the speaker's lips to obtain visual dynamic information reflecting the speaker's vocalization state. Since pure acoustic noise reduction is difficult to distinguish between the target speaker's voice and the voice of others in non-steady-state noise environments, and there is a natural temporal correspondence between lip movement and speech production, the terminal can use the extracted lip movement features to indicate the duration and intensity of the target speaker's voice in subsequent fusion noise reduction.

[0044] The terminal uses the lip key points among the key points of the speaker's face as a reference, and continuously extracts time-series images of the region of interest of the speaker's lips from the video stream captured by the binocular camera. The images of this region are scaled to ensure that the size of the lip region is consistent in each frame, and grayscale processing is performed to convert the color images into grayscale images to reduce the amount of subsequent calculations. After the above processing, the terminal obtains a normalized lip region image sequence, which records the morphological changes of the speaker's lips within a continuous time window.

[0045] Simultaneously, based on the location of lip key points among facial key points, the terminal calculates the speaker's lip opening degree (vertical distance between the upper and lower lips), lip width (horizontal distance between the left and right corners of the mouth), and lip corner displacement (offset of the corners of the mouth in the vertical and horizontal directions). These geometric parameters are then dynamically combined over time to form a lip geometric feature vector. This feature vector directly characterizes the macroscopic morphological changes of the lips with a low-dimensional representation. The terminal also inputs a normalized lip region image sequence into a lightweight convolutional network pre-deployed in the NPU. This network performs temporal feature encoding on the lip region images, extracting a lip appearance feature vector that represents the deep dynamic patterns of lip texture and shape. This appearance feature vector can capture subtle lip changes that geometric parameters cannot reflect.

[0046] The terminal aligns the geometric and apparent feature vectors of the lips temporally to ensure that the two features correspond to the lip state at the same moment. These features are then concatenated and fused to obtain the speaker's complete lip movement features, which serve as the visual input for subsequent fusion and noise reduction. Through the complementary fusion of geometric and apparent features, the terminal obtains a comprehensive lip movement representation that includes both macroscopic lip deformation and microscopic texture dynamics. This provides a reliable visual-speech activity prior for subsequent audiovisual fusion and noise reduction, enabling the noise reduction network to accurately distinguish the target voice from interference signals based on the lip movement features. Furthermore, these lip movement features can be reused for endpoint detection in the speech recognition stage, achieving multi-stage reuse of visual features.

[0047] In one embodiment, the step of fusing and denoising based on the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition to obtain the target audio includes: A reference signal played by the terminal speaker is obtained, and the audio signal obtained from the far-field audio acquisition is adaptively filtered and echo-cancelled using the reference signal as a reference to obtain the first audio after echo cancellation. The first audio is subjected to delayed summation and fixed beamforming. The main lobe is pointed to the direction corresponding to the sound source localization result. The optimal weighting vector of each element channel of the microphone array is calculated using the minimum variance distortionless response criterion. The optimal weighting vector is applied to the audio after fixed beamforming to perform adaptive beamforming, and spatially filtered audio is obtained. Statistical spectrum estimation is performed on the spatially filtered audio to reduce noise and obtain the second audio after front-end noise reduction. The lip movement features are fused with the acoustic features extracted from the second audio and then applied to the second audio to obtain the target audio.

[0048] After acquiring lip movement features and multiple audio signals from far-field audio acquisition, the terminal needs to extract the pure target human voice from the multiple signals through fusion and noise reduction. In a bank branch environment, the digital human voice broadcast by the terminal's own speaker can enter the microphone through spatial feedback, directly interfering with the customer's voice pickup. At the same time, there are also steady-state noises such as air conditioning fans and banknote counting machines, as well as non-steady-state interferences such as sudden voices from others in the branch. The terminal first acquires the reference signal currently being played by the speaker, which is a digital copy of the content broadcast by the speaker. Based on this reference signal, the terminal uses an adaptive filter to simulate the path of the echo signal after propagation in space and entering the microphone, generating an echo estimation signal. This echo estimation signal is then subtracted from the original multiple audio signals obtained from far-field audio acquisition, thereby eliminating the interference of the terminal's own broadcast sound on the customer's voice pickup and obtaining the first audio after echo cancellation.

[0049] After echo cancellation, the terminal needs to perform spatial filtering on the first audio signal to further suppress interference from directions other than the target direction. Specifically, based on the previously obtained sound source localization results, the terminal applies delayed summation fixed beamforming to the first audio signal. This involves aligning the received signals from each array element according to the theoretical propagation delay corresponding to the sound source direction and then superimposing them. This enhances the in-phase superposition of signal components from the target direction, while weakening signal components from other directions due to phase inconsistency. This directs the main lobe of the pickup beam towards the direction corresponding to the sound source localization results. Based on this, the terminal calculates the optimal weighting vector for each channel of the microphone array using the minimum variance distortionless response criterion. This criterion requires minimizing the array output power while ensuring distortion-free signal in the target direction. The terminal applies this optimal weighting vector to the audio signal after fixed beamforming for adaptive beamforming. This ensures the beam maintains distortion-free response in the target direction while forming adaptive nulls in the interference direction, thereby suppressing directional interference in space and obtaining spatially filtered audio. The optimal weighting vector refers to a set of weighting coefficients that minimize the output power while maintaining distortion-free response in the target direction under the minimum variance distortionless response criterion. The terminal performs statistical spectrum estimation noise reduction on the spatially filtered audio, that is, estimates the noise power spectrum based on the spectral statistical characteristics of the noise segment, and subtracts the noise estimate from the current signal spectrum to suppress steady-state noise and slowly varying interference such as air conditioner fans and banknote counters in the environment, and obtains the second audio after front-end noise reduction.

[0050] Because pure acoustic noise reduction struggles to effectively distinguish between target human voice and noise when their spectra overlap, especially when sudden speech from others shares similar spectral characteristics with the target human voice, it is prone to false suppression or residual noise. Therefore, the terminal fuses previously extracted lip movement features of the speaker with acoustic features extracted from the second audio. Lip movement features reflect the speaker's lip movement at every moment and can serve as a visual criterion for the presence of the target human voice, while acoustic features contain spectral energy distribution information from the second audio. Using the speaker's vocalization time and intensity indicated by the lip movement features as prior information, the terminal guides differentiated processing of different time-frequency components in the acoustic features. Specifically, it implements protective enhancement of the target human voice component in the time-frequency region corresponding to the lip movement features, and suppresses non-target human voice or pure noise components in the time-frequency region unrelated to the lip movement features. This fine-tuning is achieved through the guidance of lip movement information. The result of this fusion is applied to the second audio, resulting in fidelity enhancement of the human voice portion and further suppression of interference, ultimately outputting the target audio. During the fusion process, the terminal also protects the processing intensity to suppress human voice distortion caused by excessive suppression. When the confidence of visual information is insufficient, the terminal automatically reduces the fusion processing weight to fall back to a more conservative noise reduction mode.

[0051] Further, the step of fusing the lip movement features with the acoustic features extracted from the second audio and then applying this fusion process to the second audio to obtain the target audio includes: Log-Mel spectrum is extracted from the second audio as acoustic features. The acoustic features are then processed by the convolutional layer and temporal modeling layer of the acoustic branch of the audiovisual fusion noise reduction network to obtain the acoustic embedding. The lip movement features are processed by the convolutional layer and temporal modeling layer of the visual branch of the audiovisual fusion noise reduction network to obtain the visual embedding. The acoustic embedding and the visual embedding are spliced ​​together after temporal alignment and then fused with attention-gated weighted fusion. The time-frequency masking coefficient is output by the fusion temporal network. The time-frequency masking coefficient is applied to the complex spectrum of the second audio to obtain the target audio.

[0052] After obtaining the second audio after front-end noise reduction and the speaker's lip movement characteristics, the terminal needs to refine the second audio through an audiovisual fusion noise reduction network to solve the problem that pure acoustic noise reduction is difficult to effectively distinguish between the target human voice and noise when their spectra overlap. The terminal extracts the logarithmic Mel spectrum from the second audio as an acoustic feature. The logarithmic Mel spectrum maps the linear frequency of the audio signal to the Mel frequency scale that conforms to the characteristics of human hearing and takes the logarithmic energy, which can preserve the spectral envelope information of the speech in a relatively compact feature dimension. The terminal inputs the acoustic features into the acoustic branch of the audiovisual fusion noise reduction network. This acoustic branch includes a convolutional layer and a temporal modeling layer. The convolutional layer performs local time-frequency pattern extraction on the input log-Mel spectrum to capture the local correlation structure of the speech signal in the time and frequency dimensions. The temporal modeling layer performs temporal dependency modeling based on the feature sequence output by the convolutional layer to capture the long-range dynamic change pattern between speech frames. After the above processing by the acoustic branch, the acoustic embedding is output. This acoustic embedding represents the acoustic pattern information of the mixture of target human voice and noise in the second audio in the form of a feature vector.

[0053] Simultaneously, the terminal inputs the previously obtained lip movement features of the speaker into the visual branch of the audiovisual fusion and noise reduction network. This visual branch also includes convolutional layers and temporal modeling layers. The convolutional layers extract spatial patterns from the lip movement features, while the temporal modeling layer models the temporal dynamics of lip movements between consecutive frames. After processing by the visual branch, a visual embedding is output. This visual embedding represents the dynamic pattern information of the speaker's lip vocalization activities in the form of feature vectors. The acoustic embedding and visual embedding represent the speech-related information at the same moment from the acoustic modality and visual modality, respectively, and the two correspond in the temporal dimension.

[0054] The terminal strictly aligns the acoustic and visual embeddings temporally to ensure that acoustic and visual features from the same time index correspond to each other. Then, the aligned acoustic and visual embeddings are concatenated to form a fused feature vector containing both acoustic and visual information. The terminal applies an attention-gated weighted fusion mechanism to this fused feature vector. The attention gating dynamically evaluates the importance of each modality feature using learned weight coefficients. During periods of high target voice intensity, the contribution of the visual embedding to the fusion result is increased to strengthen the protection of the target voice by the noise reduction network. During periods of low confidence in visual information or insignificant lip movement features, the weight of the visual embedding is reduced to avoid introducing erroneous noise reduction due to incorrect visual information. Thus, the fusion weights of the two modalities are adaptively adjusted according to the current acoustic and visual conditions. After attention-gated weighted fusion, the terminal inputs the fused features into the fusion temporal network. This fusion temporal network further models the weighted fused feature sequence in the time dimension and outputs the time-frequency masking coefficient corresponding to each time-frequency unit of the second audio. The masking coefficient is a continuous value between 0 and 1, which represents the probability or confidence level of the corresponding time-frequency unit belonging to the target human voice.

[0055] The terminal applies the time-frequency masking coefficient point-by-point to the complex spectrum of the second audio. Specifically, it multiplies each time-frequency unit in the complex spectrum by the corresponding masking coefficient. Time-frequency units with masking coefficients close to 1 are identified as dominated by the target human voice and preserved, while those close to 0 are identified as dominated by noise and suppressed. After point-by-point application of the masking coefficient, the terminal restores the processed complex spectrum to a time-domain waveform through inverse time-frequency transformation, obtaining the final target audio. For example, during periods when the speaker is continuously speaking and their lip movements are active, the visual embedding confidence of the visual branch output is high. Attention gating increases the visual contribution weight, and the fusion temporal network outputs a masking coefficient close to 1 for time-frequency units synchronized with the lip movement rhythm during this period, ensuring the complete preservation of the target human voice. Conversely, during periods when the call-up announcement suddenly sounds and the speaker's lips are still, attention gating reduces the visual weight, and the fusion temporal network outputs a masking coefficient close to 0 for time-frequency units of the announcement sound without lip movement, effectively suppressing the announcement sound.

[0056] By extracting audiovisual features in two channels, aligning temporally, and weighting with attention gating, the network can accurately determine which components belong to the target human voice and protect them, and which components belong to the interference signal and suppress them, even in non-steady-state noise scenarios with overlapping spectra. This achieves a balance between noise reduction depth and human voice fidelity, providing high-quality target audio for subsequent speech recognition.

[0057] In one embodiment, performing speech recognition on the target audio to obtain target text includes: The target audio is pushed to the input of the speech recognition engine in a circular buffer to obtain streaming audio data to be recognized. Acoustic speech activity detection is performed based on the acoustic energy characteristics of the target audio to obtain acoustic endpoint candidates; The acoustic endpoint candidates are reviewed based on the visual speech activity detection results of the lip movement features to obtain the sentence endpoints; Using the endpoints of the statement as the statement boundaries, the posterior probability of the phonemes is output from the audio data to be identified through a streaming acoustic model, and the target text is obtained by performing a beam search decoding on a weighted finite state converter graph through a decoder.

[0058] After acquiring the target audio, the terminal needs to convert the audio signal into text data through a speech recognition engine for subsequent semantic parsing module to understand the business. Although noise in the bank branch environment has been significantly suppressed after double noise reduction, residual non-steady-state interference may still cause endpoint misjudgment in acoustic speech activity detection, and the accuracy of endpoint location directly affects the completeness of the recognition results.

[0059] The terminal continuously pushes the target audio to the input of the speech recognition engine in a circular buffer with a buffer depth of 200 milliseconds. This buffering mechanism can absorb data jitter caused by front-end signal processing and avoid additional delays caused by data accumulation. After receiving the data, the engine obtains the streaming input audio data to be recognized. During the recognition process, the terminal performs acoustic speech activity detection based on the spectral energy characteristics of the target audio and the modeled speech probability, initially determines the start and end times of the speech segment, and outputs acoustic endpoint candidates.

[0060] Because relying solely on acoustic information can easily lead to endpoint misjudgment in high-noise environments, the terminal simultaneously performs visual speech activity detection on the previously extracted speaker's lip movement features. This means that the presence of speech is determined by whether the speaker's lips are active, as indicated by the lip movement features, thus obtaining a visual determination of the start and end of speech. The terminal then uses the visual speech activity detection results to review the acoustic endpoint candidates. When the acoustically detected speech start and end times match the lip movement features, the terminal confirms the endpoint as valid and uses it as a sentence endpoint. When the two are inconsistent, the terminal corrects the acoustic endpoint based on the visual lip movement information.

[0061] By employing a lip movement feature verification mechanism, the terminal avoids the problem of premature truncation or excessive prolongation of the final sound due to acoustic misjudgment in noisy scenarios such as sudden announcements or calls for numbers, ultimately obtaining reliable sentence endpoints. After obtaining the sentence endpoints, the terminal uses these endpoints as sentence boundaries and outputs the posterior probability of phonemes frame by frame through a streaming acoustic model deployed on the terminal. A decoder then performs beam search decoding on a pre-built weighted finite state converter graph. This decoding process integrates the acoustic model score and the language model score to score candidate word sequences, searching for and outputting the word sequence with the highest score as the target text. For example, when a speaker says "What is the interest rate of a structured deposit?", the sentence endpoints obtained by the terminal through lip movement feature verification precisely correspond to the actual start and end times of this sentence, avoiding text loss caused by endpoint truncation, and ultimately outputting the complete target text for subsequent semantic analysis.

[0062] By reusing lip movement features in the endpoint detection stage of speech recognition, the terminal can still ensure the accurate delineation of sentence endpoints even in high-noise environments, thereby ensuring the integrity and accuracy of the target text and enabling semantic parsing to obtain reliable text input.

[0063] In one embodiment, determining the business skill to be executed and interacting with the target user based on the semantic analysis result of the target text according to a preset financial terminology database includes: The terminology database is configured by loading a pre-set financial terminology database into the multimodal voice interaction terminal at a bank branch. The financial terminology database includes terminology entries, synonymous colloquial expressions, and associated business skill identifiers. The target text is subjected to intent recognition and slot extraction to obtain a structured semantic result containing intent tags and slot fields; The intent tags and slot fields in the structured semantic results are compared with the matching conditions of each business skill in the skill library of the bank branch multimodal voice interaction terminal, and the matching score of each business skill is calculated. The business skill with the highest matching score that exceeds the preset execution threshold is identified as the business skill to be executed, and the user interacts with the target user through the business skill to be executed.

[0064] After obtaining the target text, the terminal needs to convert the natural language expression in text form into structured business instructions and match them with corresponding service skills to complete the business interaction with the target user. Since the recognized text output in step five is an unstructured natural language expression, and may contain non-standard colloquial terms, synonyms, or abbreviations, directly executing business operations based on it is prone to errors. Furthermore, general semantic understanding models lack domain knowledge of financial professional terms such as structured deposits, large-denomination certificates of deposit, and net asset value-based wealth management, making it difficult to accurately interpret the customer's true business needs.

[0065] The terminal pre-builds and stores a financial terminology database for banking operations. This database is stored locally on the terminal as a hot-updateable structured file and can be configured and expanded via a cloud platform. It covers business lines such as deposits, wealth management, loans, credit cards, account services, fund transfers, precious metals, and foreign exchange. It includes standard terminology entries for each business line, synonymous colloquial expressions, and business skill identifiers associated with each terminology entry. After loading this financial terminology database, the terminal obtains a database configuration. This configuration includes the mapping relationship from colloquial expressions to standardized terms and the association information between terms and business skills, providing a domain knowledge foundation for subsequent intent parsing and skill matching.

[0066] The terminal then performs intent recognition and slot extraction on the target text. Intent recognition is used to determine the type of business requirement of the customer's current interaction, while slot extraction is used to extract specific business parameters related to the business requirement from the text, such as product type, amount, term, and currency. After the above processing, the terminal outputs a structured semantic result containing intent tags and slot fields. This structured semantic result converts natural language text into machine-readable key-value pairs. The terminal then compares the intent tags and slot fields in the structured semantic result with the matching conditions of each business skill in the skill library. The skill library contains multiple professional skills covering service scenarios such as intelligent greeting guidance, self-service business consultation, product introduction, branch navigation, and queuing. The skill library also supports users to customize skill configurations through a cloud platform, enabling the addition and deletion of skills, editing of scripts, and customization of semantic scenarios.

[0067] The terminal calculates a matching score for each business skill based on the comparison results. The matching score reflects the degree of fit between the skill and the customer's semantic needs. The terminal identifies the business skill with the highest matching score that exceeds the preset execution threshold as the business skill to be executed. When the matching score is within the preset follow-up question range, the terminal generates a clarifying follow-up question to guide the customer to supplement the necessary information. When no skill's matching score exceeds the execution threshold, the terminal switches to a fallback script or a human customer service guidance process.

[0068] For example, when a customer says the colloquial expression "save money," the terminal, after intent recognition, categorizes it as a deposit business intent. The slot retrieval result is empty, and skill matching identifies the deposit consultation skill as the skill to be executed. The terminal then broadcasts the current deposit product information through the speaker and displays a deposit interest rate table on the screen. Simultaneously, the recognized text and structured semantic results are reported to the digital human master control via RS232 serial port. As another example, when a customer says "What is the interest rate of a structured deposit?", the terminal, after intent recognition, outputs the intent label "interest rate query" and the slot containing the product type "structured deposit." Skill matching successfully matches the structured semantic result with the interest rate query skill, and the terminal then triggers the interest rate query skill to broadcast the expected interest rate range of the current structured deposit product to the customer.

[0069] Through the comprehensive application of domain knowledge guidance provided by the financial terminology database, structured transformation achieved by intent recognition and slot extraction, and skill matching mechanism, the terminal can accurately understand customers' colloquial business requests and trigger the correct service skills, effectively avoiding business processing errors caused by terminology recognition errors or semantic misunderstandings, and improving the accuracy of digital human services and customer satisfaction.

[0070] Reference Figure 2 The diagram shown is a functional module schematic of the multimodal voice interaction device 100 for bank branches according to this application.

[0071] According to the functions implemented, the multimodal voice interaction device 100 for bank branches includes a positioning module 110, a detection module 120, a determination module 130, an extraction module 140, a fusion module 150, a recognition module 160, and an interaction module 170. These modules can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0072] In this embodiment, the functions of each module / unit are as follows: Positioning module 110: used to perform far-field audio acquisition and sound source localization in the bank branch environment, and obtain sound source localization results; Detection module 120: used to perform face detection and distance estimation operations on the network environment to obtain face targets; Determining module 130: used to perform spatiotemporal alignment and verification operations between the sound source localization result and the face target to determine the speaker in the face target; Extraction module 140: used to extract lip movement features from the speaker's lips to obtain the speaker's lip movement features; Fusion module 150: used to perform fusion and noise reduction based on the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition to obtain the target audio; Recognition module 160: used to perform speech recognition on the target audio to obtain target text; Interaction module 170: Used to determine the business skills to be executed and interact with the target user based on the semantic analysis results of the target text according to the preset financial terminology database.

[0073] The specific implementation of the multimodal voice interaction device for bank branches in this application is largely the same as the specific implementation of the multimodal voice interaction method for bank branches described above, and will not be repeated here.

[0074] Reference Figure 3The diagram shown is a schematic representation of a preferred embodiment of the electronic device of this application.

[0075] The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. The memory 113 is used to store computer programs, such as a multimodal voice interaction program for bank branches; Figure 3 Only an electronic device having a processor 111, a communication interface 112, a memory 113 and a communication bus 114 is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0076] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the multimodal voice interaction method for bank branches provided in any of the foregoing method embodiments, including: Far-field audio acquisition and sound source localization were performed on the environment of bank branches to obtain sound source localization results; Face detection and distance estimation operations are performed on the network environment to obtain face targets; The sound source localization result is spatiotemporally aligned and verified with the face target to determine the speaker in the face target; Lip movement features of the speaker are extracted; The target audio is obtained by fusing and denoising the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition. Perform speech recognition on the target audio to obtain the target text; Based on the semantic analysis results of the target text using a pre-set financial terminology database, the business skills to be executed and the interaction with the target user are determined.

[0077] For a detailed description of the above steps, please refer to the flowchart of the above embodiment of the multimodal voice interaction method for bank branches.

[0078] Furthermore, this application also proposes a computer-readable storage medium that is both non-volatile and volatile. This computer-readable storage medium is any one or any combination of several of the following: hard disk, multimedia card, SD card, flash memory card, SMC, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area. The program storage area stores a multimodal voice interaction program for bank branches. When executed by a processor, the multimodal voice interaction program for bank branches performs the following operations: Far-field audio acquisition and sound source localization were performed on the environment of bank branches to obtain sound source localization results; Face detection and distance estimation operations are performed on the network environment to obtain face targets; The sound source localization result is spatiotemporally aligned and verified with the face target to determine the speaker in the face target; Lip movement features of the speaker are extracted; The target audio is obtained by fusing and denoising the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition. Perform speech recognition on the target audio to obtain the target text; Based on the semantic analysis results of the target text using a pre-set financial terminology database, the business skills to be executed and the interaction with the target user are determined.

[0079] The specific implementation of the computer-readable storage medium in this application is largely the same as the specific implementation of the multimodal voice interaction method for bank branches described above, and will not be repeated here.

[0080] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware simulation platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0082] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal voice interaction method for bank branches, characterized in that, The method includes: Far-field audio acquisition and sound source localization were performed on the environment of bank branches to obtain sound source localization results; Face detection and distance estimation operations are performed on the network environment to obtain face targets; The sound source localization result is spatiotemporally aligned and verified with the face target to determine the speaker in the face target; Lip movement features of the speaker are extracted; The target audio is obtained by fusing and denoising the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition. Perform speech recognition on the target audio to obtain the target text; Based on the semantic analysis results of the target text using a pre-set financial terminology database, the business skills to be executed and the interaction with the target user are determined.

2. The multimodal voice interaction method for bank branches as described in claim 1, characterized in that, The step of performing spatiotemporal alignment and verification between the sound source localization result and the facial target to determine the speaker in the facial target includes: The sound source localization results are interpolated on the time axis to align with the sampling time of the video frames of each face target, and the azimuth angle of each face target in the image pixel coordinate system is mapped to the microphone array coordinate system to obtain the aligned face target and azimuth information. The corresponding orientation deviation tolerance is calculated based on the distance from each face target to the terminal, where the terminal is a multimodal voice interaction terminal in a bank branch. Calculate the original deviation between the sound source localization result and the orientation angle of each face target, and select face targets whose original deviation is less than the corresponding orientation deviation tolerance and whose faces are facing the terminal as candidate targets. If there is only one candidate target, then that candidate target will be the speaker. If there are multiple candidate targets, the one with the smallest distance to the terminal among all candidate targets will be selected as the speaker.

3. The multimodal voice interaction method for bank branches as described in claim 1, characterized in that, The step of extracting lip movement features from the speaker's lips to obtain the speaker's lip movement features includes: Based on the key facial points of the speaker, the region of interest (ROI) image of the lips is cropped from the speaker's facial image. The ROI image of the lips is then normalized and grayscaled to obtain a normalized lip region image sequence. Based on the position of the lip key points among the facial key points, the speaker's lip opening degree, lip width, and lip corner displacement are calculated to obtain the speaker's lip geometric feature vector; The normalized lip region image sequence is input into a lightweight convolutional network for temporal feature encoding to obtain the lip appearance feature vector of the speaker. The lip geometric feature vector and the lip apparent feature vector are aligned temporally and then concatenated to obtain the speaker's lip movement features.

4. The multimodal voice interaction method for bank branches as described in claim 1, characterized in that, The step of fusing and denoising based on the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition to obtain the target audio includes: A reference signal played by the terminal speaker is obtained, and the audio signal obtained from the far-field audio acquisition is adaptively filtered and echo-cancelled using the reference signal as a reference to obtain the first audio after echo cancellation. The first audio is subjected to delayed summation and fixed beamforming. The main lobe is pointed to the direction corresponding to the sound source localization result. The optimal weighting vector of each element channel of the microphone array is calculated using the minimum variance distortionless response criterion. The optimal weighting vector is applied to the audio after fixed beamforming to perform adaptive beamforming, and spatially filtered audio is obtained. Statistical spectrum estimation is performed on the spatially filtered audio to reduce noise and obtain the second audio after front-end noise reduction. The lip movement features are fused with the acoustic features extracted from the second audio and then applied to the second audio to obtain the target audio.

5. The multimodal voice interaction method for bank branches as described in claim 4, characterized in that, The step of fusing the lip movement features with the acoustic features extracted from the second audio and then applying this fusion to the second audio to obtain the target audio includes: Log-Mel spectrum is extracted from the second audio as acoustic features. The acoustic features are then processed by the convolutional layer and temporal modeling layer of the acoustic branch of the audiovisual fusion noise reduction network to obtain the acoustic embedding. The lip movement features are processed by the convolutional layer and temporal modeling layer of the visual branch of the audiovisual fusion noise reduction network to obtain the visual embedding. The acoustic embedding and the visual embedding are spliced ​​together after temporal alignment and then fused with attention-gated weighted fusion. The time-frequency masking coefficient is output by the fusion temporal network. The time-frequency masking coefficient is applied to the complex spectrum of the second audio to obtain the target audio.

6. The multimodal voice interaction method for bank branches as described in claim 1, characterized in that, Performing speech recognition on the target audio to obtain the target text includes: The target audio is pushed to the input of the speech recognition engine in a circular buffer to obtain streaming audio data to be recognized. Acoustic speech activity detection is performed based on the acoustic energy characteristics of the target audio to obtain acoustic endpoint candidates; The acoustic endpoint candidates are reviewed based on the visual speech activity detection results of the lip movement features to obtain the sentence endpoints; Using the endpoints of the statement as the statement boundaries, the posterior probability of the phonemes is output from the audio data to be identified through a streaming acoustic model, and the target text is obtained by performing a beam search decoding on a weighted finite state converter graph through a decoder.

7. The multimodal voice interaction method for bank branches as described in claim 1, characterized in that, The step of determining the business skills to be executed and interacting with the target user based on the semantic analysis results of the target text according to the preset financial terminology database includes: The terminology database is configured by loading a pre-set financial terminology database into the multimodal voice interaction terminal at a bank branch. The financial terminology database includes terminology entries, synonymous colloquial expressions, and associated business skill identifiers. The target text is subjected to intent recognition and slot extraction to obtain a structured semantic result containing intent tags and slot fields; The intent tags and slot fields in the structured semantic results are compared with the matching conditions of each business skill in the skill library of the bank branch multimodal voice interaction terminal, and the matching score of each business skill is calculated. The business skill with the highest matching score that exceeds the preset execution threshold is identified as the business skill to be executed, and the user interacts with the target user through the business skill to be executed.

8. A multimodal voice interaction device for bank branches, characterized in that, The device includes: Positioning module: used to perform far-field audio acquisition and sound source localization in the bank branch environment, and obtain sound source localization results; Detection module: used to perform face detection and distance estimation operations on the network environment to obtain face targets; Determination module: used to perform spatiotemporal alignment and verification operations between the sound source localization result and the face target to determine the speaker in the face target; Extraction module: used to extract lip movement features from the speaker's lips to obtain the speaker's lip movement features; Fusion module: used to fuse and reduce noise based on the lip movement features and the acoustic features of the audio signal acquired from the far-field audio acquisition to obtain the target audio; Recognition module: used to perform speech recognition on the target audio to obtain the target text; Interaction module: Used to determine the business skills to be executed and interact with the target user based on the semantic analysis results of the target text according to the preset financial terminology database.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the multimodal voice interaction method for bank branches as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal voice interaction method for bank branches as described in any one of claims 1 to 7.

Images