Bank counter talkback noise reduction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a bank counter talkback noise reduction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring video information and audio information of a client; the video information and the audio information are acquired by a target camera which is arranged on a counter and is internally provided with a microphone; performing feature extraction on the video information and the audio information, and performing cross attention feature fusion on the extracted audio features and video features to obtain audition action features; and determining a target audio feature containing the context information, and inputting the target audio feature and the audio-visual action feature into a cross-context attention feature fusion module for noise reduction processing to obtain a target audio. By adopting the technical scheme of the embodiment of the invention, environmental noise and other customer sounds are filtered through a voice noise reduction algorithm based on audio-video cooperation, the business handling efficiency of a teller is improved, and the customer journey experience of a user is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a bank counter intercom noise reduction method, device, electronic equipment and storage medium. Background Art

[0002] Bank counters are the specific places where bank customers conduct business and have always played an important role in the development history of banks. Although the emergence of new operating services such as super counters has replaced some of the functions of traditional counters, some special businesses, such as large-value transfers, bank card loss reports, loans and other important businesses, still need to be handled at traditional counters.

[0003] At present, the distance between bank counters in offline branches is generally small, and the distance between counters and waiting customers is relatively small. When customers communicate with tellers through traditional microphones, the voices received by tellers are generally mixed with environmental noises such as the voices of non-target customers, which will affect the efficiency of communication between tellers and customers, and then affect the efficiency of tellers in handling business, and reduce customer experience. Offline branches usually set up partitions on the counters or add independent reception rooms to solve the above problems, but the cost is relatively high, and there are many branches, so the feasibility is relatively poor.

[0004] Therefore, how to reduce noise in the communication between tellers and customers is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] The embodiments of the present invention provide a bank counter intercom noise reduction method, device, electronic device and storage medium to filter environmental noise and other customer sounds, improve the efficiency of tellers in handling business, and enhance the customer journey experience of users.

[0006] In a first aspect, an embodiment of the present invention provides a bank counter intercom noise reduction method, comprising:

[0007] Acquire video information and audio information of the client; the video information and audio information are acquired by a target camera with a built-in microphone installed on the counter;

[0008] Extracting features from the video information and the audio information, and performing cross-attention feature fusion on the extracted audio features and video features to obtain audition action features;

[0009] The target audio features containing context information are determined, and the target audio features and audio-visual action features are input into a cross-context attention feature fusion module for noise reduction processing to obtain the target audio.

[0010] In a second aspect, an embodiment of the present invention further provides a bank counter intercom noise reduction device, comprising:

[0011] An audio and video information acquisition module is used to acquire video information and audio information from the client; the video information and audio information are acquired by a target camera with a built-in microphone installed on the counter;

[0012] An audio and video feature cross-fusion module is used to extract features from the video information and the audio information, and to perform cross-attention feature fusion on the extracted audio features and video features to obtain audition action features;

[0013] The audio and video feature cross-context attention fusion module is used to determine the target audio features containing context information, and input the target audio features and audio-visual action features into the cross-context attention feature fusion module for noise reduction processing to obtain the target audio.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] A storage device for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the bank counter intercom noise reduction method described in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the bank counter intercom noise reduction method described in any embodiment of the present invention is implemented.

[0019] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the bank counter intercom noise reduction method as described in any embodiment of the present invention.

[0020] The embodiment of the present invention provides a bank counter intercom noise reduction method, device, electronic device and storage medium, which obtains the video information and audio information of the client; the video information and audio information are obtained by a target camera with a built-in microphone set on the counter; the video information and audio information are feature extracted, and the extracted audio features and video features are cross-attention feature fused to obtain audition action features; the target audio features containing context information are determined, and the target audio features and audio-visual action features are input into a cross-context attention feature fusion module for noise reduction processing to obtain target audio. The technical solution of the embodiment of the present invention is adopted to filter environmental noise and other customer sounds through a voice noise reduction algorithm based on audio-visual collaboration, thereby improving the efficiency of tellers in handling business and increasing the user's customer journey experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts. In the drawings:

[0022] Figure 1 It is a flow chart of a bank counter intercom noise reduction method provided in an embodiment of the present invention;

[0023] Figure 2 It is a schematic diagram of the relationship between facial movements and speech waveforms during speaking provided in an embodiment of the present invention;

[0024] Figure 3 It is a structural schematic diagram of a bank counter intercom noise reduction system provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of an intelligent bank counter provided in an embodiment of the present invention;

[0026] Figure 5 is a flow chart of another bank counter intercom noise reduction method provided in an embodiment of the present invention;

[0027] Figure 6 1 is a schematic diagram of a network structure of speech noise reduction based on audio and video information collaboration provided in an embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of the structure of a cross-context attention feature fusion module provided in an embodiment of the present invention;

[0029] Figure 8 It is a structural schematic diagram of a bank counter intercom noise reduction device provided in an embodiment of the present invention;

[0030] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0032] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0033] Among them, the acquisition, storage, use and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components or models may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0035] Embodiment 1

[0036] Figure 1 This is a flow chart of a bank counter intercom noise reduction method provided in an embodiment of the present invention. This embodiment can be applied to the situation of bank counter intercom noise reduction in banking business scenarios. The method of this embodiment can be executed by a bank counter intercom noise reduction device, which can be implemented in hardware and / or software. The device can be configured in a server for bank counter intercom noise reduction. The method specifically includes the following steps:

[0037] S110: Obtain video information and audio information from the client.

[0038] Among them, traditional speech noise reduction algorithms include adaptive filtering algorithms, spectral subtraction, and neural network-based speech noise reduction algorithms. These methods generally use the difference between human voice and environmental noise to reduce the noise of speech. Therefore, for some mixed voice situations, these algorithms generally show poor performance. In real life, when listening to a person's speech, humans usually observe their facial movements and mouth movements. Combining this visual information can better understand the target person's speech. This is because vocalization and mouth movements are closely linked. This is audio-visual synergy; see Figure 2 , showing the relationship between facial movements and speech waveforms during human speech.

[0039] Therefore, the embodiment of the present invention proposes a speech noise reduction algorithm based on audio-visual collaboration. This algorithm can well capture the facial movement information of the target speaker, and use this information to assist the speech noise reduction task, and has better noise reduction performance when dealing with scenarios with mixed multi-voice voices. This method is exactly what is needed in banking business scenarios. Banking business scenarios often face a mixture of different human voices, so some noise reduction algorithms are needed to filter out environmental human voice noise. For example, when handling business at the counter, the teller only needs the voice of the target customer, and can use the speech noise reduction algorithm based on audio-visual collaboration to filter out environmental noise and other customer voices, thereby improving the efficiency of the teller in handling business and increasing the user's customer journey experience.

[0040] See also Figure 3 The embodiment of the present invention provides a bank counter intercom noise reduction system, which is mainly composed of a data acquisition end and a voice noise reduction processing unit. The data acquisition end mainly includes a data acquisition unit, which collects the voice information and facial movement information of the client. The voice noise reduction processing unit uses a voice noise reduction algorithm based on audio-visual collaboration to denoise the collected voice based on the collected voice information and the customer's lip movement information, and finally plays the denoised voice to the bank teller.

[0041] See also Figure 4 The data collection end is composed of two speakers 1 and 2, a camera with a microphone 3, and a teller microphone 4. The data collection end mainly uses a camera with a microphone to collect data, obtain the video information and audio information of the client, and provide data support for the subsequent voice noise reduction processing unit. The video information and audio information of the client are obtained by a target camera with a built-in microphone set on the counter.

[0042] Optionally, in the embodiment of the present invention, only the audio information and video information of the client need to be obtained, and the existing camera at the counter can be directly reused, so there is no need to add additional equipment. It is only necessary to integrate the voice noise reduction processing unit into the computer without too much additional cost.

[0043] S120, extracting features from the video information and the audio information, and performing cross-attention feature fusion on the extracted audio features and video features to obtain audition action features.

[0044] Among them, the attention mechanism is a technology that imitates cognitive attention in artificial neural networks. This mechanism can enhance the weights of certain parts of the neural network input data, while weakening the weights of other parts, so as to focus the network's attention on the most important small part of the data. Generally, the soft weights are calculated using a query-key mechanism. In an embodiment of the present invention, the extracted audio features and video features are cross-attentive feature fusion, and the cross-attention feature fusion may refer to cross-fusion calculation of some features in the video with features in the audio. For example, when determining the audio features, some features in the video are fused into the audio for calculation, and when determining the video features, some features in the audio are cross-fused with features in the video for calculation, and the audio features and video features obtained by the cross-fusion calculation are spliced ​​to obtain the audition action features.

[0045] S130, determining target audio features containing context information, and inputting the target audio features and audio-visual action features into a cross-context attention feature fusion module for noise reduction processing to obtain target audio.

[0046] Among them, noise reduction processing can refer to separating the speaker's voice, which means separating the target speaker's voice from a piece of speech and removing other noises and other speakers' voices. From the perspective of task development, it can be roughly divided into two stages, one is pure audio-based speaker separation; the other is speaker separation based on audio-visual fusion multimodal in combination with visual information. In the embodiment of the present invention, the speech is denoised in a multimodal manner based on audio-video fusion.

[0047] Optionally, by determining the target audio features containing context information, the target audio features and the audio-visual action features are input into the cross-context attention feature fusion module for noise reduction processing, so as to obtain the target audio with background noise removed. That is, in the banking business scenario, the ambient noise around the target customer is removed, and only the target customer's voice is retained to communicate with the teller.

[0048] The embodiment of the present invention provides a bank counter intercom noise reduction method, which obtains the video information and audio information of the client; the video information and audio information are obtained by a target camera with a built-in microphone set on the counter; the video information and audio information are feature extracted, and the extracted audio features and video features are cross-attention feature fused to obtain audition action features; the target audio features containing context information are determined, and the target audio features and audio-visual action features are input into a cross-context attention feature fusion module for noise reduction processing to obtain target audio. The technical solution of the embodiment of the present invention is adopted to filter environmental noise and other customer sounds through a voice noise reduction algorithm based on audio-visual collaboration, thereby improving the efficiency of tellers in handling business and increasing the user's customer journey experience.

[0049] Embodiment 2

[0050] Figure 5 The flowchart of a bank counter intercom noise reduction method provided in an embodiment of the present invention. The embodiment of the present invention further optimizes the above embodiment on the basis of the above embodiment, and the embodiment of the present invention can be combined with various optional solutions in one or more of the above embodiments. Figure 5 As shown, the bank counter intercom noise reduction method provided in the embodiment of the present invention may include the following steps:

[0051] S510: Obtain video information and audio information of the client.

[0052] Among them, the embodiment of the present invention proposes a speech noise reduction algorithm based on the coordination of audio and video information. This algorithm can well capture the facial action information of the target speaker, use this information to assist the speech noise reduction task, and has a good noise reduction performance in the scene of processing multiple voices. Therefore, the embodiment of the present invention needs to obtain the video information and audio information of the client.

[0053] S520: Segment the video information to obtain a continuous facial video frame sequence.

[0054] Among them, see Figure 6 The present invention designs an audio-visual action extractor module. The function of this module is to extract more reliable visual information. The principle is to use visual information and audio information at the same time to obtain the audio-visual action information of the target person, so that non-voice actions such as chewing and yawning can be judged. There are two inputs to this module, one is a facial video frame sequence V. First, the acquired video information is segmented, such as the acquired video information is segmented according to the specification of 25 frames per second, and a continuous facial video frame sequence V is obtained as input.

[0055] S530, performing Mel-frequency cepstral coefficient processing on the audio information to obtain audio features.

[0056] Among them, another input of the audio-visual action extractor module is the audio feature. The acquired audio information is processed by Mel-Frequency Cepstral Coefficients (MFCCs) to obtain the audio feature S.

[0057] S540: Input the facial video frame sequence and the audio features into an audio-visual action information extractor to extract audio timing features and video timing features.

[0058] The audio-visual action information extractor is responsible for extracting the audio-visual action information of the target speaker from the video and audio. Specifically, the input audio features S and the face video frame sequence V are passed through two different audio feature extractors and visual feature extractors to obtain audio timing features and video timing features; the audio timing features and video timing features represent the audio-related activities and visual-related motion information, respectively.

[0059] The embodiment of the present invention utilizes the synergy between facial visual information and audio information during speaking to design an audiovisual action information extractor. Compared with traditional visual feature information, the audiovisual action information can provide more reliable facial action information.

[0060] S550, cross-attention feature fusion is performed on the extracted audio timing features and video timing features to obtain audition action features.

[0061] Since the speaking event is an audiovisual synchronization event, the visual features contain some information required by the audio features, and the audio features also contain information required by the visual features. Based on this, the embodiment of the present invention designs a cross-attention mechanism in the time dimension to dynamically describe this audiovisual feature interaction.

[0062] As an optional but non-limiting implementation, the extracted audio timing features and video timing features are cross-attentionally fused to obtain the audition action features, including but not limited to steps A1-A2:

[0063] Step A1: linearly transform the extracted audio timing features and video timing features to obtain audio linear features and video linear features.

[0064] Step A2: cross-attention feature fusion is performed on the audio linear features and the video linear features to obtain fused target audio linear features and target video linear features.

[0065] Step A3: concatenate the fused audio linear features and video linear features to obtain the audition action features.

[0066] Among them, the cross-attention module performs a linear transformation on the original input. In the embodiment of the present invention, the extracted audio timing features and video timing features are respectively subjected to linear transformation (denoted as W) to obtain Query, Key and Value (denoted as Q, K and V), and Q and K are dot-multiplied to perform cross-attention feature fusion on the audio linear features and the video linear features to obtain the fused target audio linear features and the target video linear features.

[0067] The audio linear features and the video linear features are cross-attentionally fused to obtain the fused target audio linear features and the target video linear features, which are expressed as:

[0068]

[0069] in, represents the target audio linearity characteristic, Q a , K a and V a represents the linear characteristics of the audio obtained by linear transformation, represents the linear features of the target video, Q v , K v and V v represents the linear features of the video obtained by linear transformation, is a constant.

[0070] The above formulas respectively show how to encode video features into audio features and apply audio information to visual features, calculate the correlation between the two, and then perform scale transformation, masking and softmax operations to calculate the interdependence between the two. It is to prevent the dot product of Q and K from being too large.

[0071] Optionally, the extracted audio timing features and video timing features are linearly transformed, which can be expressed as:

[0072]

[0073] In the above formula, the subscripts a and v represent the features obtained from audio and visual respectively. After adding the cross attention mechanism, the correlation between different modalities can be well captured. The final feature is of After splicing operation, the audition action feature F can be obtained av .

[0074] S560: Determine target audio features containing context information, and input the target audio features and audio-visual action features into a cross-context attention feature fusion module for noise reduction processing to obtain target audio.

[0075] When the audio intermediate feature F obtained by the encoder e , after Na convolutions in the separation network, the feature F containing context information is obtained ′ e , at this time, it is necessary to combine the audio-visual action feature F av Combined. av It is a context information feature containing the speaker's facial movements. Therefore, in order to better combine these two features, the embodiment of the present invention designs a Cross-Context Attention Fusion (CCAF) module.

[0076] As an optional but non-limiting implementation, determining the target audio features containing context information, and inputting the target audio features and the audition action features into the cross-context attention feature fusion module for noise reduction processing to obtain the target audio, includes but is not limited to steps B1-B4:

[0077] Step B1: Input the audio information into the audio encoder to obtain the audio intermediate features.

[0078] Step B2: Convolution processing is performed on the intermediate audio features to determine target audio features containing context information.

[0079] Step B3: Input the target audio features and the audition action features into the cross-context attention feature fusion module and the convolution module for noise reduction processing to obtain an audio mask.

[0080] Step B4: Obtain target audio based on the audio mask and the audio intermediate features.

[0081] Among them, see Figure 7 The cross-context attention feature fusion module is mainly used to utilize audio-visual action information. The module consists of a self-attention module, a cross-attention module and a multi-layer perceptron containing two linear layers. The self-attention module is similar to the cross-attention module. The difference is that the two different features are input in the cross-attention module, while the input in the self-attention module is the same. The CCAF module can effectively fuse the audio-visual action information of the target speaker for better speech noise reduction.

[0082] After processing the target audio features and the audition action features using the convolution module and CCAF module, an audio mask M can be obtained. The audio mask contains the information of the clean audio. The audio decoder D se The audio mask and the output of the audio encoder (audio intermediate features) will be used to obtain the final speaker speech separation result y eThe process can be described by the following formula:

[0083] M=separator(CCAF(CB(F e ),F av ))

[0084] y e = decoder(M·F e )

[0085] Among them, y e represents the target audio, M represents the audio mask, and F e represents the audio intermediate feature, F av Indicates the characteristics of the audition action; CB(F e ) represents the convolution processing of the intermediate audio features to obtain the target audio features; CCAF(·) represents the cross-context attention feature fusion module.

[0086] At this point, the target audio to be separated can be obtained. In addition, the algorithm was tested on a public data set in the embodiment of the present invention. Compared with the mainstream noise reduction algorithm of the same period, the SI-SDR and PESQ evaluation indicators of the present invention were improved by about 10% and 8% respectively. In addition, the present invention was also tested on a real data set and obtained good results.

[0087] The embodiment of the present invention provides a bank counter intercom noise reduction method for situations where customers and tellers cannot communicate in a friendly manner due to noisy human voices in offline outlets. By applying the voice noise reduction algorithm to the intercom equipment of the bank counter, the noise in communication can be effectively reduced, work efficiency can be improved, and customer experience can be enhanced. A voice noise reduction algorithm based on the collaboration of audio and video information is used to solve the problem that traditional voice noise reduction algorithms cannot handle multi-voice noise. An audio-visual action information extractor that uses both video information and audio information is designed to solve the shortcoming of unreliable visual information in previous audio-visual multimodal noise reduction algorithms, so as to improve the denoising effect when processing multi-voice mixed noise, and can more accurately extract the voice of the target customer, adding more convenience to the teller's service.

[0088] Embodiment 3

[0089] Figure 8 This is a schematic diagram of the structure of a bank counter intercom noise reduction device provided in an embodiment of the present invention. The technical solution of this embodiment can be applied to the situation of bank counter intercom noise reduction. The device can be implemented by software and / or hardware and is generally integrated on any electronic device with network communication function, including but not limited to: servers, computers, personal digital assistants and other devices. Figure 8As shown, the bank counter intercom noise reduction device provided in this embodiment may include: an audio and video information acquisition module 810, an audio and video feature cross-fusion module 820, and an audio and video feature cross-context attention fusion module 830; wherein,

[0090] The audio and video information acquisition module 810 is used to acquire the video information and audio information of the client; the video information and audio information are acquired by a target camera with a built-in microphone installed on the counter;

[0091] The audio and video feature cross fusion module 820 is used to extract features from the video information and the audio information, and to perform cross attention feature fusion on the extracted audio features and video features to obtain audition action features;

[0092] The audio and video feature cross-context attention fusion module 830 is used to determine the target audio features containing context information, and input the target audio features and audio-visual action features into the cross-context attention feature fusion module for noise reduction processing to obtain the target audio.

[0093] Based on the above embodiment, optionally, the audio and video feature cross fusion module is specifically used to:

[0094] Segment the video information to obtain a continuous sequence of face video frames;

[0095] The audio information is processed by Mel cepstral coefficients to obtain audio features;

[0096] Inputting the face video frame sequence and audio features into the audio-visual action information extractor to extract audio timing features and video timing features;

[0097] The extracted audio timing features and video timing features are cross-attentionally fused to obtain the audition action features.

[0098] Based on the above embodiment, optionally, the audio and video feature cross fusion module is specifically used to:

[0099] The extracted audio timing features and video timing features are linearly transformed to obtain audio linear features and video linear features;

[0100] Performing cross-attention feature fusion on the audio linear features and the video linear features to obtain fused target audio linear features and target video linear features;

[0101] The fused audio linear features and video linear features are spliced ​​together to obtain the audition action features.

[0102] Based on the above embodiment, optionally, the audio and video feature cross fusion module is specifically used to:

[0103] The audio linear features and the video linear features are cross-attentionally fused to obtain the fused target audio linear features and the target video linear features, which are expressed as:

[0104]

[0105] in, represents the target audio linearity characteristic, Q a , K a and V a represents the linear characteristics of the audio obtained by linear transformation, represents the linear features of the target video, Q v , K v and V v represents the linear features of the video obtained by linear transformation, is a constant.

[0106] Based on the above embodiment, optionally, the audio and video feature cross-context attention fusion module is specifically used for:

[0107] Input the audio information into the audio encoder to obtain the audio intermediate features;

[0108] Performing convolution processing on the intermediate audio features to determine target audio features containing context information;

[0109] Inputting the target audio features and the audition action features into a cross-context attention feature fusion module and a convolution module for noise reduction processing to obtain an audio mask;

[0110] The target audio is obtained according to the audio mask and the intermediate audio features.

[0111] Based on the above embodiment, optionally, the audio and video feature cross-context attention fusion module is specifically used for:

[0112] According to the audio mask and the audio intermediate features, the target audio is obtained, which is expressed as:

[0113] M=separator(CCAF(CB(F e ),F av ))

[0114] y e = decoder(M·F e )

[0115] Among them, y e represents the target audio, M represents the audio mask, and Fe represents the audio intermediate feature, F av Indicates the characteristics of the audition action; CB(F e ) represents the convolution processing of the intermediate audio features to obtain the target audio features; CCAF(·) represents the cross-context attention feature fusion module.

[0116] The bank counter intercom noise reduction device provided in the embodiment of the present invention can execute the bank counter intercom noise reduction method provided in any embodiment of the present invention, and has the corresponding functions and beneficial effects of executing the bank counter intercom noise reduction method. For detailed process, please refer to the relevant operations of the bank counter intercom noise reduction method in the aforementioned embodiment.

[0117] Embodiment 4

[0118] Fig. 9 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0119] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0120] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a bank counter intercom noise reduction method.

[0122] In some embodiments, the bank counter intercom noise reduction method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the bank counter intercom noise reduction method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the bank counter intercom noise reduction method by any other appropriate means (for example, by means of firmware).

[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0128] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0129] Embodiment 5

[0130] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the bank counter intercom noise reduction method provided in any embodiment of the present application.

[0131] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A bank counter intercom noise reduction method, characterized in that: The method comprises: Obtaining video information and audio information from the client; the video information and audio information are obtained by a target camera with a built-in microphone installed on the counter; Extracting features from the video information and the audio information, and cross-attentionally fusing the extracted audio features and video features to obtain audition action features; The target audio features containing context information are determined, and the target audio features and audio-visual action features are input into a cross-context attention feature fusion module for noise reduction processing to obtain the target audio.

2. The method according to claim 1, characterized in that The step of extracting features from the video information and the audio information, and performing cross-attention feature fusion on the extracted audio features and video features to obtain audition action features includes: Segment the video information to obtain a continuous sequence of face video frames; The audio information is processed by Mel cepstral coefficients to obtain audio features; Inputting the face video frame sequence and audio features into the audio-visual action information extractor to extract audio timing features and video timing features; The extracted audio timing features and video timing features are cross-attentionally fused to obtain the audition action features.

3. The method according to claim 2, characterized in that The extracted audio timing features and video timing features are cross-attentionally fused to obtain audition action features, including: The extracted audio timing features and video timing features are linearly transformed to obtain audio linear features and video linear features; Performing cross-attention feature fusion on the audio linear features and the video linear features to obtain fused target audio linear features and target video linear features; The fused audio linear features and video linear features are spliced ​​together to obtain the audition action features.

4. The method according to claim 3, characterized in that The cross-attention feature fusion of the audio linear features and the video linear features to obtain fused target audio linear features and target video linear features includes: The audio linear features and the video linear features are cross-attentionally fused to obtain the fused target audio linear features and the target video linear features, which are expressed as: in, represents the target audio linearity characteristic, Q a , K a and V a represents the linear characteristics of the audio obtained by linear transformation, represents the linear features of the target video, Q v , K v and V v represents the linear features of the video obtained by linear transformation, is a constant.

5. The method according to claim 1, characterized in that The step of determining target audio features containing context information, and inputting the target audio features and audition action features into a cross-context attention feature fusion module for noise reduction processing to obtain target audio includes: Input the audio information into the audio encoder to obtain the audio intermediate features; Performing convolution processing on the intermediate audio features to determine target audio features containing context information; Inputting the target audio features and the audition action features into a cross-context attention feature fusion module and a convolution module for noise reduction processing to obtain an audio mask; The target audio is obtained according to the audio mask and the intermediate audio features.

6. The method according to claim 5, characterized in that The step of obtaining the target audio according to the audio mask and the audio intermediate feature includes: According to the audio mask and the audio intermediate features, the target audio is obtained, which is expressed as: M=separator(CCAF(CB(F e ), F av )) y e =decoder(M·F e ) Among them, y e represents the target audio, M represents the audio mask, and F e represents the audio intermediate feature, F av Indicates the characteristics of the audition action; CB(F e ) represents the convolution processing of the intermediate audio features to obtain the target audio features; CCAF(·) represents the cross-context attention feature fusion module.

7. A bank counter intercom noise reduction device, characterized in that: The device comprises: An audio and video information acquisition module is used to acquire video information and audio information from the client; the video information and audio information are acquired by a target camera with a built-in microphone installed on the counter; An audio and video feature cross-fusion module is used to extract features from the video information and the audio information, and to perform cross-attention feature fusion on the extracted audio features and video features to obtain audition action features; The audio and video feature cross-context attention fusion module is used to determine the target audio features containing context information, and input the target audio features and audio-visual action features into the cross-context attention feature fusion module for noise reduction processing to obtain the target audio.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the bank counter intercom noise reduction method described in any one of claims 1-6.

9. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the bank counter intercom noise reduction method as described in any one of claims 1-6 when executed by a computer processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the bank counter intercom noise reduction method according to any one of claims 1 to 6 is implemented.

Images