A bimodal adaptive speech recognition method for high-noise aquatic product transaction scenarios

Abstract

The application is suitable for the field of artificial intelligence and speech recognition technology, and provides a bimodal adaptive speech recognition method for high-noise aquatic product transaction scenes, which collects audio and visual lip shape signals and extracts features first; realizes bimodal feature fusion through a multi-level bidirectional cross attention of noise perception, dynamically adjusts the mode weight based on the real-time signal-to-noise ratio; realizes multi-dialect incremental learning through a cumulative multi-task flexible weight consolidation strategy, realizes multi-market safe cooperative training in combination with federated learning; finally, through the aquatic product transaction knowledge graph and multi-signal entity linking, the spoken expression is mapped to structured transaction instructions. The application significantly improves the recognition accuracy in a high-noise environment, adapts to multi-dialect and vertical industry scenes, and has a broad industrialization prospect.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and speech recognition technology, specifically a dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios. Background Technology

[0002] Speech recognition technology has made significant progress in quiet environments, but it still faces significant challenges in complex industrial settings (such as seafood trading markets). These scenarios typically have the following characteristics: extremely high ambient noise levels (70-90 decibels) and multiple non-stationary noise sources; the use of local dialects by employees (such as Minnan and Cantonese), which are insufficiently supported by general speech recognition models; and the inclusion of highly specialized industry terms and colloquial expressions in trading language, which are difficult for general semantic understanding models to parse.

[0003] In the field of bimodal recognition, Audio-Visual Speech Recognition (AVSR) technology improves robustness by fusing audio and lip-sync visual signals. However, existing methods are mainly geared towards controlled, low-noise laboratory environments, and their application in noisy industrial environments remains insufficient. This is mainly manifested in the following ways: modal fusion weights are typically fixed or only subject to simple linear adjustments, lacking adaptive dynamic adjustment capabilities based on real-time signal-to-noise ratio; cross-attention mechanisms use fixed temperature scaling factors, failing to adapt to noise conditions; and there is a trade-off between model complexity and real-time performance. In the field of dialect recognition, transfer learning and online adaptive methods based on Wav2Vec 2.0 have made progress on a few dialects, but incremental learning for multiple coastal dialects still suffers from catastrophic forgetting. Existing elastic weight consolidation methods only consider protecting knowledge from the most recently learned task; when learning multiple dialects sequentially, the dialect knowledge learned earlier will gradually degrade. Furthermore, data privacy protection and federated learning efficiency issues in multi-market scenarios have not been effectively resolved.

[0004] Therefore, in view of the above situation, there is an urgent need to provide a dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios to overcome the shortcomings in current practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios, effectively solving the problems mentioned in the background art.

[0006] This invention is implemented as follows: a dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios, comprising the following steps: S1. Audio signal acquisition and preprocessing: Audio signals are acquired using a microphone array and noise reduction is performed to output the noise-reduced audio feature sequence. S2. Visual lip shape signal acquisition and feature extraction: Capture facial video sequences using a camera, extract the region of interest (ROI) of the lips, and use a 3D convolutional neural network to extract spatiotemporal features of the ROI to obtain a visual feature sequence. S3. Noise-aware multi-level progressive bimodal feature fusion: The audio feature sequence and the visual feature sequence are respectively input into the corresponding encoder network. A noise-aware adaptive temperature scaling bidirectional cross-attention mechanism is introduced in multiple preset levels of the encoder. The cross-attention output is adaptively integrated into the original features of the corresponding modality through gated residual connection to obtain the fused audio features and visual features. S4. Dynamic modal weight allocation based on signal-to-noise ratio: The signal-to-noise ratio of the current environment is estimated in real time, and after time-series smoothing, it is input into a differentiable generalized asymmetric Sigmoid mapping function to calculate the dynamic fusion weights of audio and visual modalities; based on the dynamic fusion weights, the fused audio features and visual features are weighted and summed to obtain the final multimodal fusion features. S5. Adaptive speech recognition for coastal dialects: Speech recognition is performed based on the multimodal fusion features to obtain text results; a cumulative multi-task elastic weight consolidation strategy is adopted, and a time decay regularization term for all learned dialect tasks is introduced into the loss function to achieve adaptive recognition of multiple coastal dialects. S6. Knowledge Graph-Based Transaction Semantic Understanding: Utilizing a pre-constructed knowledge graph in the aquatic product transaction domain, a multi-signal confidence scoring method integrating semantic embedding, contextual probability, graph structure importance, and dialect phonological similarity is employed to link entities in the identified text. After completing relation extraction and intent classification, the structured transaction instructions obtained through mapping are output.

[0007] As a further aspect of the present invention: the microphone array in step S1 includes 8 MEMS microphones; The noise reduction process is as follows: The acquired audio signal is input into a noise suppression module based on a deep neural network for noise reduction processing.

[0008] As a further aspect of the present invention: the camera in step S2 supports 120fps acquisition, and the three-dimensional convolutional neural network consists of 5 three-dimensional convolutional layers and 3 fully connected layers, with the kernel size of each three-dimensional convolutional layer being 3×3×3.

[0009] As a further aspect of the present invention: in step S3, audio features are used as key-value pairs of query and visual features for cross-attention calculation, and visual features are used as key-value pairs of query and audio features for cross-attention calculation. The specific implementation of the noise-aware adaptive temperature scaling bidirectional cross-attention mechanism and the gated residual connection is as follows: In each fusion layer of the encoder, the audio feature matrix is ​​transformed linearly to generate a query matrix, and the visual feature matrix is ​​transformed linearly to generate a key matrix and a value matrix, respectively. A noise-aware adaptive temperature coefficient is introduced, and the cross-attention output from audio to visual direction is calculated using a scaled dot product attention method. The attention score is obtained by multiplying the query matrix and the transpose of the key matrix, dividing by the product of the square root of the key vector dimension and the adaptive temperature coefficient, and then multiplying by the value matrix after softmax normalization. The adaptive temperature coefficient is generated by mapping the signal-to-noise ratio estimated in the current frame through a ReLU activation function after linear transformation, ensuring that the temperature coefficient is not lower than 1. The weights and biases of the linear transformation are independent learnable parameters for each fusion layer. The cross-attention output from the visual to the audio direction is calculated in the same way, and its corresponding adaptive temperature coefficient is independently parameterized. The cross-attention outputs from each direction are integrated into the original features of the corresponding modality through a gated residual connection mechanism: After concatenating the original audio features and the cross-attention output along the feature dimension, an element-wise gating factor is generated by linear transformation and sigmoid activation function. The cross-attention output is then gated and modulated by element-wise multiplication and added to the original features as a residual. After layer normalization, the updated audio features are output. The visual features are updated by gating residuals in the same way.

[0010] As a further aspect of the present invention: the specific implementation of the dynamic modal weight allocation in step S4 is as follows: The signal-to-noise ratio (SNR) of the current audio frame is estimated in real time using the spectrum analysis module, resulting in the instantaneous SNR. The instantaneous signal-to-noise ratio is then subjected to exponential moving average time-series smoothing. ; in, For smoothing coefficients, For the first Signal-to-noise ratio after smoothing. for The smooth signal-to-noise ratio at any given moment; Smoothed signal-to-noise ratio Calculate the visual modality weights by taking a differentiable generalized asymmetric sigmoid mapping function as input: ; in, A learnable temperature parameter for controlling the steepness of the weight transition. To control the learnable offset parameter of the weight transition center point, Learnable shape parameters for controlling the asymmetry of the weighted transition curve; Audio modal weights ; The dynamic fusion weights are applied to the weighted summation of the fused features to obtain the final multimodal fusion features: ; in, The audio features are those obtained after multi-level cross-attention fusion. These are visual features resulting from multi-level cross-attention fusion.

[0011] As a further aspect of the present invention: the specific implementation of the cumulative multi-task elastic weight consolidation strategy in step S5 is as follows: When the first is introduced in sequence When fine-tuning the model using dialect data, a method specific to all dialects is introduced into the loss function. Cumulative regularization terms for a learned dialect task: ; in, For the current number The loss in dialect recognition task, For the first The first task corresponding to the learned dialect Fisher information content of each parameter For the model number The current values ​​of the parameters, To complete the first The parameter values ​​saved after learning a dialect task; Regularization weights for each learned task A time decay strategy is adopted for allocation, which is calculated by multiplying the basic regularization coefficient by a negative exponential decay function with respect to the temporal distance of the task, so that the dialects learned recently are better protected by knowledge, and the decay rate is an adjustable hyperparameter.

[0012] As a further aspect of the present invention: step S5 also includes a federated adaptive training mechanism based on dialect distribution similarity weighting, specifically implemented as follows: Terminal devices deployed in multiple aquatic product trading markets each use local dialect speech data for model training, and only the model parameter updates are uploaded to the central aggregation server, while the original data does not leave the domain; Differential privacy protection with sensitivity calibration is applied before parameter upload, which means adding a Gaussian noise vector to the local model parameters of each market. The vector has a mean of zero, a variance proportional to the product of the square of the privacy noise coefficient and the square of the gradient clipping threshold, and is inversely proportional to the amount of local data. The central aggregation server uses a federated average algorithm based on dialect distribution similarity weighting to aggregate model parameters. The aggregation weight of each market is obtained by normalizing the product of its local data volume and dialect distribution similarity weight. The dialect distribution similarity weight is defined as the negative exponent of the mean Jensen-Shannon divergence of dialect phoneme distribution among markets.

[0013] As a further aspect of the present invention: the specific implementation of the multi-signal fusion confidence scoring method in step S6 is as follows: For each text segment detected in the identified text Retrieve a set of candidate entities from the aquatic product trading knowledge graph, and for each candidate entity... Calculate the confidence score for multi-signal fusion: ; in, The cosine similarity between the candidate entity embedding vector and the mentioned text embedding vector is... For context-based windows Entity conditional probability, Assess the importance of entities within the knowledge graph. For phonological similarity scoring based on dialect phoneme confusion matrix, , , , Learnable weight coefficients to satisfy normalization constraints; The candidate entity with the highest confidence score is selected as the link result. Based on the linked entities, relationship extraction and intent classification are completed to generate a structured trading instruction containing elements of variety, specification, quality, and price.

[0014] As a further aspect of the present invention: the knowledge graph in the aquatic product trading field described in step S6 is constructed by combining top-down ontology design with bottom-up instance filling. The ontology layer defines four core entity types: variety entity, specification entity, quality entity, and price entity, and defines the synonym relationships, hierarchical relationships, attribute association relationships, and co-occurrence relationships between entities.

[0015] As a further aspect of the present invention: step S5 also includes a speaker adaptation step: When a new user uses the app for the first time, the app reads aloud preset commonly used trading phrases. Based on the collected samples, the model's attention layer is fine-tuned using a low-rank decomposition strategy, and the fine-tuned personalized parameters are stored on the user's local device.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Improve speech recognition accuracy in high-noise environments This invention constructs a multi-level progressive audio-video fusion architecture, introducing a noise-aware adaptive temperature scaling bidirectional cross-attention mechanism and gated residual connections at multiple encoder levels to achieve deep adaptive feature interaction between audio and visual modalities. Unlike existing cross-attention mechanisms that use a fixed temperature scaling factor, the adaptive temperature coefficient designed in this invention is generated driven by real-time signal-to-noise ratio (SNR), automatically reducing the temperature coefficient under high noise conditions to sharpen the attention distribution and make cross-modal feature alignment more accurate. The gated residual connection mechanism further introduces a learnable gating factor, element-wise controlling the degree to which the cross-attention output is integrated into the original features, avoiding the unconditional injection of low-quality cross-modal information under noisy conditions. Regarding modal weight allocation, this invention uses a dynamic weight allocation scheme based on a differentiable generalized asymmetric Sigmoid mapping function instead of the traditional piecewise linear mapping. By introducing learnable asymmetric shape parameters, the transition rate of weights in high and low SNR regions can be independently optimized. Combined with exponential moving average temporal smoothing processing, the weight transition is smooth and differentiable throughout, supporting end-to-end optimization. When the signal-to-noise ratio (SNR) is below 0dB, the system automatically increases the visual modality weight to over 70%, fully utilizing lip-sync visual information to compensate for severe degradation of audio information. At the hardware level, it employs an 8-microphone array beamforming directional sound pickup technology and a noise suppression module based on a deep neural network, effectively suppressing non-human voice frequency noise by 25-30dB. The visual channel uses a 120fps high-frame-rate binocular camera combined with a 3D convolutional neural network for spatiotemporal feature extraction, exhibiting excellent robustness against lighting changes and water mist interference. This hardware and software collaborative solution enables the system to maintain over 90% recognition accuracy even in high-noise aquatic product trading environments of 70-90dB, effectively overcoming the technical bottleneck of severe performance degradation in existing speech recognition technologies under noisy industrial environments.

[0017] 2. Achieve synergistic optimization of multi-dialect adaptive recognition and data privacy protection. This invention designs a cumulative multi-task elastic weight consolidation incremental learning framework. Unlike existing standard elastic weight consolidation methods that only protect knowledge from the most recently learned task, this invention introduces a cumulative regularization term covering all learned dialect tasks into the loss function and employs a time decay strategy to differentiate the regularization weights for each task. This provides stronger knowledge protection for recently learned dialects, effectively mitigating catastrophic forgetting at each stage when learning multiple dialects sequentially. The framework supports adaptive recognition of eight major coastal dialects, including Minnan, Cantonese, and Wu, and constructs a dialect terminology database containing specialized terms for aquatic product trading. Regarding federated collaborative training, this invention proposes an improved federated averaging algorithm based on dialect distribution similarity weighting. Market similarity weights are obtained by calculating the Jensen-Shannon divergence of dialect phoneme distributions among markets, allowing markets with similar dialect distributions to receive higher contribution weights in model aggregation. This effectively improves the low convergence efficiency of the standard federated averaging algorithm under non-independent and identically distributed data conditions. Meanwhile, a differential privacy protection mechanism for sensitivity calibration is introduced, and the noise intensity is inversely correlated with the amount of local data in each market. This protects data privacy while minimizing the impact on model accuracy, providing a feasible technical path for distributed deployment.

[0018] 3. Achieve accurate semantic understanding and intent recognition in aquatic product trading scenarios. This invention constructs a dedicated knowledge graph for the aquatic product trading field, covering over 200 aquatic product varieties, multi-dimensional specifications, freshness and quality grades, and price ranges. It also defines a rich semantic relationship network including synonyms, hierarchical relationships, and attribute associations. In the entity linking stage, this invention proposes a multi-signal fusion confidence scoring method. This method comprehensively scores candidate entities based on four dimensions: semantic embedding similarity, contextual conditional probability, knowledge graph structural importance, and dialect phonological similarity. Compared to traditional methods relying solely on string matching or single semantic similarity, this method achieves higher linking accuracy. In particular, by introducing a phonological similarity scoring signal based on a dialect phoneme confusion matrix, it demonstrates significantly stronger robustness in handling speech recognition-to-transcription errors caused by dialect accents. Based on this knowledge graph and the entity linking results, the system can accurately map frequently occurring colloquial, dialectal, and simplified expressions in aquatic product trading into standardized structured trading instructions. For example, when a trader says in a dialect, "Give me two boxes of large yellow croaker, fresh, at wholesale price," the system can accurately identify the species (large yellow croaker), specifications (boxed), quality requirements (fresh), and price intention (wholesale price), and generate a structured transaction order containing these elements. This technology effectively compensates for the shortcomings of general semantic understanding models in vertical industry applications, providing key technical support for the digitalization and intelligentization of aquatic product trading processes, and can significantly reduce transaction disputes and economic losses caused by communication errors. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The present invention will be further explained below with reference to specific embodiments.

[0023] Please see Figure 1 The present invention provides a dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios, comprising the following steps: S1: Audio signal acquisition and preprocessing A customized microphone module with IP67 protection is deployed next to the cashier or electronic scale at the trading stall. This module integrates eight MEMS microphones arranged in a uniform circular array with a diameter of 60mm. The system employs a beamforming algorithm based on Minimum Variance Distortionless Response (MVDR). The optimal spatial filter weight vector is obtained by inverting the noise covariance matrix and multiplying it by the steering vector in the target direction, followed by normalization. This algorithm controls the main lobe width within ±30°, achieving directional microphone pickup of the target speaker and effectively suppressing ambient noise from the sides and rear.

[0024] The acquired multi-channel audio signals first undergo acoustic echo cancellation to eliminate interference from the speaker's voice on the microphone signal. Subsequently, the multi-channel signals are beamformed and synthesized into a single-channel enhanced signal. This single-channel enhanced signal is then fed into a deep neural network-based denoising model using a U-Net architecture, which performs mask estimation of the signal's amplitude spectrum in the frequency domain. The denoising model learns a time-frequency mask, and by element-wise multiplying the network-estimated mask value with the short-time Fourier transform coefficients of the noisy signal, the estimated clean speech spectrum is obtained. The model achieves 25-30 dB noise suppression in non-human voice frequency bands (such as the 100-500 Hz low-frequency band where mechanical noise is concentrated and the 2000-8000 Hz high-frequency band where water flow noise is concentrated). The denoised audio signal is output at a 16 kHz sampling rate and 16-bit quantization precision, and after Mel-frequency cepstral coefficient (MFCC) feature extraction, an 80-dimensional audio feature sequence is generated with a frame shift of 10 ms.

[0025] The IP67 protection rating ensures that the pickup module can work stably for a long time in the humid and foggy environment of aquatic product trading, avoiding equipment failure caused by moisture intrusion or dust accumulation, and guaranteeing the reliability and durability of the system in actual industrial environments.

[0026] S2: Visual lip shape signal acquisition and feature extraction A binocular camera module is deployed facing the speaker at the trading booth. The two cameras are 65mm apart, with a resolution of 1920×1080, and support a high frame rate of 120fps. This high frame rate capability allows the system to capture subtle changes in rapid lip movements, which is particularly crucial for fast-paced speech scenarios common in aquatic product trading, such as rapid price quotations and listing of consecutive varieties.

[0027] The system first employs a face detection algorithm based on a multi-task cascaded convolutional neural network (MTCNN) to locate the facial region of the target speaker in real time, and then uses a 68-keypoint face alignment algorithm to accurately locate the lip region. The region of interest in the lips is cropped into a 96×96 pixel grayscale image sequence. This image sequence is then fed into a three-dimensional convolutional neural network (3D-CNN) for spatiotemporal feature extraction. The 3D-CNN consists of 5 three-dimensional convolutional layers and 3 fully connected layers. The kernel size of each three-dimensional convolutional layer is 3×3×3 (time×height×width), capturing the dynamic change trend of lip shape in the temporal dimension and extracting lip shape features in the spatial dimension. The calculation process of each three-dimensional convolutional layer is as follows: a sliding window convolution operation is performed on the input features in the spatial and temporal dimensions; the weights of the convolutional kernel are weighted and summed with the feature values ​​of the corresponding input region, a bias term is added, and then a ReLU linear rectified activation function is used for nonlinear mapping to output the feature representation of that layer. To enhance the robustness of visual features to common lighting changes in aquatic product trading environments (such as the transition between indoor and outdoor lighting, and the flickering of fluorescent lights), data augmentation strategies are employed during the training phase, including random brightness adjustments (±40%), contrast variations (0.6-1.4 times), and Gaussian noise injection. Furthermore, depth information from the binocular cameras is used to assist in determining the distance between the speaker and the camera; an early warning mechanism is triggered when the working distance exceeds the effective range of 0.5-2 meters. The 3D-CNN outputs a 512-dimensional visual feature sequence, with the frame rate aligned with the audio feature sequence.

[0028] S3: Multi-level progressive bimodal fusion for noise perception Both the audio encoder and the visual encoder adopt a 12-layer Transformer architecture with a hidden layer dimension of 512 and 8 attention heads. Noise-aware, temperature-scaling-adaptive bidirectional cross-attention fusion modules are introduced in layers 3, 6, 9, and 12 of the encoder.

[0029] Taking the nth layer fusion as an example, let the audio and visual features of this layer be real-valued matrices of different dimensions, where the sequence lengths of the audio and visual sequences correspond to their respective time steps. The cross-attention calculation process from audio to vision is as follows: Audio features are transformed linearly to generate a query matrix, and visual features are transformed linearly to generate a key matrix and a value matrix, respectively. The key vector dimension of each attention head is the hidden layer dimension divided by the number of attention heads (i.e., 64 dimensions).

[0030] A noise-aware adaptive temperature coefficient is introduced, which is generated by linear transformation and ReLU activation based on the real-time estimated signal-to-noise ratio of the current frame, ensuring that the temperature coefficient is not lower than 1. The linear transformation parameters of each fusion layer are independently learnable. When the signal-to-noise ratio is low, the temperature coefficient approaches 1, the attention distribution is sharper, and the model focuses on the most discriminative local information in the visual features. When the signal-to-noise ratio is high, the temperature coefficient increases, the attention distribution is smoother, and broader cross-modal information interaction is allowed.

[0031] The cross-attention output is calculated based on the adaptive temperature coefficient: the attention score is obtained by dividing the scaled dot product of the query and key by the temperature coefficient, normalizing by softmax, and then multiplying it by the value matrix. The cross-attention output in the visual-to-audio direction is calculated in the same way, with its corresponding adaptive temperature coefficient independently parameterized.

[0032] A gated residual connection mechanism is employed to adaptively integrate the cross-attention output into the original features: the original features and the cross-attention output are concatenated along the feature dimension, and a linear transformation and sigmoid activation are performed to generate an element-wise gating factor. The cross-attention output is then gated and modulated, added to the original feature residual, and subjected to layer normalization. This gating factor independently controls the degree of cross-modal information integration for each feature dimension, effectively suppressing the introduction of low-quality information under noisy conditions. The visual features are then updated using the same gated residual mechanism.

[0033] By introducing the aforementioned fusion modules at layers 3, 6, 9, and 12 of the encoder, the system achieves progressive integration of modal information at different abstraction levels: low-level fusion captures the underlying temporal alignment relationship between acoustic features and lip dynamics, while high-level fusion integrates semantic-level cross-modal complementary information, thereby achieving more comprehensive cross-modal feature utilization than single-layer fusion.

[0034] S4: Standardized Structured Output and Secure Transmission The system analyzes the short-time Fourier transform spectrum of the current audio frame and uses a posterior signal-to-noise ratio estimation algorithm based on minimum mean square error to calculate the environmental signal-to-noise ratio in real time. The estimated instantaneous signal-to-noise ratio is then smoothed using an exponential moving average time-series smoothing process, i.e., with a smoothing coefficient... (Corresponding to an equivalent time constant of 500ms) The smoothed signal-to-noise ratio of the previous frame and the instantaneous signal-to-noise ratio of the current frame are weighted and summed to obtain the smoothed signal-to-noise ratio. The temporal smoothing process can eliminate the drastic fluctuations in weights caused by instantaneous noise spikes and improve the stability of weight allocation.

[0035] The signal-to-noise ratio (SNR) of the current audio frame is estimated in real time using the spectrum analysis module, resulting in the instantaneous SNR. The instantaneous signal-to-noise ratio is then subjected to exponential moving average time-series smoothing. ; in This is a smoothing coefficient used to control the weighting ratio between historical and current values. The signal-to-noise ratio after smoothing (the first one) time), For the previous moment ( The smoothed signal-to-noise ratio (SNR) can be obtained by weighting the historical smoothed values ​​and the current instantaneous values. This eliminates the weight fluctuations caused by instantaneous noise spikes, making the SNR estimation more stable.

[0036] Smoothed signal-to-noise ratio Calculate the visual modality weights by taking a differentiable generalized asymmetric sigmoid mapping function as input: ; in, A learnable temperature parameter (initialized to 0.5) is used to control the steepness of the weight transition. The learnable offset parameter for controlling the weight transition center point is initialized to 5.0, corresponding to a weight transition center at a signal-to-noise ratio of 5 dB. Learnable shape parameters (initialized to 1.0) are used to control the asymmetry of the weighted transition curve. When it degenerates into the standard Sigmoid function; when In low signal-to-noise ratio (SNR) regions, the weight transition is steeper, enabling the system to switch to the visual modality-dominated mode more quickly in scenarios with sudden increases in noise. Audio modality weights. .

[0037] When the signal-to-noise ratio is higher than 10dB Approaching 0, it is dominated by audio modes; when the signal-to-noise ratio is between 0-10dB, and Smooth transition; when the signal-to-noise ratio is below 0dB, Approaching 1, dominated by the visual modality ( ).

[0038] The weight allocation result is applied to the weighted summation of the fused features to obtain the final multimodal fused features. ,in The audio features are those obtained after multi-level cross-attention fusion. These are the visual features obtained after multi-level cross-attention fusion. The fused features are then fed into the subsequent Transformer decoder network for speech recognition.

[0039] This dynamic weight allocation mechanism enables the system to automatically adjust the fusion strategy based on instantaneous environmental noise conditions. For example, when a large cold chain transport vehicle suddenly starts in the trading market (instantly increasing noise by 15-20dB), the system can smoothly switch weights within 200ms, maintaining the continuity and accuracy of identification.

[0040] S5: Incremental Learning and Federated Adaptation for Coastal Dialects The incremental learning process is illustrated using Minnan and Cantonese as examples. The system is based on a Wav2Vec 2.0 pre-trained model, which is first fine-tuned using 100 hours of Mandarin data to obtain the basic model.

[0041] Cumulative multi-task incremental learning phase: When fine-tuning the model by sequentially introducing Minnan (20 hours) and Cantonese (15 hours), a cumulative multi-task elastic weight consolidation strategy is adopted. When sequentially introducing the Tth dialect data for model fine-tuning, a cumulative regularization term for all T-1 learned dialect tasks is introduced into the loss function: ; in For the current number The loss in dialect recognition task, For the first The first task corresponding to the learned dialect Fisher information content of each parameter For the model number The current values ​​of the parameters, To complete the first The parameter values ​​saved after learning a dialect task; The regularization weights of each learned task A time decay strategy is adopted for allocation, which is calculated by multiplying the basic regularization coefficient by a negative exponential decay function with respect to the temporal distance of the task, so that the dialects learned recently are better protected by knowledge, and the decay rate is an adjustable hyperparameter.

[0042] Fisher's information The sensitivity of the parameters to the learned task is obtained by calculating the expected value of the squared gradient of the log-likelihood function with respect to the parameters on the validation set.

[0043] Federated learning collaborative training phase: Terminal devices in different markets (such as Fuzhou, Guangzhou, Ningbo, etc.) train their models using local dialect data. After each round of training, each terminal performs gradient pruning on the model parameters and adds differential privacy noise protection. The noise variance is directly proportional to the square of the privacy coefficient and inversely proportional to the amount of local data.

[0044] The central server uses a federated average algorithm based on dialect distribution similarity to aggregate parameters. The aggregation weight for each market is obtained by normalizing the product of the local data volume and the dialect distribution similarity weight. The similarity weight is defined as a negative exponent of the mean divergence of dialect phoneme distributions among markets, giving markets with similar dialects a higher contribution weight.

[0045] Speech Adaptation Phase: New users are introduced to 10-20 preset commonly used trading phrases upon first use. Based on these samples, a low-rank decomposition strategy is used to fine-tune the attention layer. This method decomposes weight updates into the product of two low-rank matrices, with fine-tuning parameters accounting for only 2% of the total parameters. Personalized parameters are stored on the user's local device.

[0046] The aquatic product trade terminology database is built using a combination of manual compilation and semi-automatic expansion. The initial database contains 5,000 commonly used trade terms. During system operation, new terminology candidates are automatically identified through unrecognized word detection and added after manual review. The database is updated weekly.

[0047] S6: Transaction Semantic Understanding Based on Knowledge Graph Knowledge Graph Construction: The aquatic product trading knowledge graph is constructed using a combination of top-down ontology design and bottom-up instance population. The ontology layer defines four core entity types and their attributes: Species Entities: Covering over 200 aquatic products including fish, shrimp, crabs, shellfish, and cephalopods, each species entity includes attributes such as standard name, local colloquial name (supporting mapping to multiple dialects), scientific name, and family classification. For example, the entity "large yellow croaker" is associated with the Minnan dialect colloquial names "yellow croaker" and "cucumber fish," and the Cantonese colloquial name "yellow flower," etc.

[0048] Specifications include weight specifications (such as "3-4 liang", "over 1 jin"), packaging specifications (such as "boxed", "basketed", "bulk"), and quantity units (such as "jin", "kilogram", "box").

[0049] Quality Entities: Define attributes such as freshness level (fresh, chilled, frozen, dried), quality grade (premium, grade 1, grade 2) and origin certification (wild, farmed, organic).

[0050] Price entity: Includes attributes such as price type (wholesale price, retail price, daily price), price range, and settlement method.

[0051] The ontology layer also defines the semantic relationship types between entities: synonym relationship (such as "large yellow croaker" and "yellow croaker"), hierarchical relationship (such as "fish" and "large yellow croaker"), attribute association relationship (such as the typical size of "large yellow croaker" is "3-5 two heads"), and co-occurrence relationship (common combinations of variety-specification-quality based on historical transaction data statistics).

[0052] The instance layer is populated using three data sources: first, core entities and relationships manually compiled by aquatic product industry experts, consisting of 8,000 entity nodes and 15,000 relationship edges; second, supplementary entities and relationships semi-automatically extracted from aquatic product trading industry standard documents, market announcements, and industry websites; and third, new entity candidates automatically discovered through transaction record analysis during system operation, which are then incorporated into the graph after manual review.

[0053] Multi-signal fusion entity linking: For the fused and recognized text, named entity recognition is first performed using a sequence labeling model based on conditional random fields to detect all possible entity mentions. For each detected mention, a Top-10 candidate entity set is retrieved from the knowledge graph based on edit distance and pinyin similarity. .

[0054] For each candidate entity Calculate the confidence score for multi-signal fusion: ; The specific calculation methods for the four signal components are as follows: Signal 1: Semantic Embedding Similarity The text to be mentioned Mapped to a 256-dimensional embedding vector using a text encoder shared with the knowledge graph embedding model. Calculate its embedding vector with that of the candidate entity. The cosine similarity is used to capture the semantic distance between the mentioned text and the candidate entities, and it is tolerant of transcription errors caused by dialect accents (such as transcribing "yellow flower" as "cucumber").

[0055] Signal 2: Contextual Conditional Probability Based on mentions The context window (5 words before and after) is used to calculate the generation probability of each candidate entity under the current context using a pre-trained language model. This signal uses contextual semantic constraints to eliminate ambiguity. For example, in the context of "give me two boxes of fresh daylilies", the probability of "daylilies" being linked to "large yellow croaker" is significantly higher than the probability of being linked to "daylily".

[0056] Signal 3: Importance of Knowledge Graph Structure The PageRank algorithm is used to calculate the structural importance score of each entity on the knowledge graph, reflecting the entity's core position in the aquatic product trading field. High-frequency trading commodities (such as large yellow croaker, shrimp, and swimming crab) receive higher PageRank scores due to having more relational connections, thus gaining priority in the candidate entity ranking.

[0057] Signal 4: Dialect Phonetic Similarity To address the problem of speech recognition and transcription deviation caused by dialect accents in the aquatic product trading scenario, a phonetic similarity score based on a dialect phoneme confusion matrix is designed. The mentioned text and the candidate entity name are respectively converted into pinyin sequences, and the weighted edit distance is calculated using a pre-constructed dialect phoneme confusion matrix, where the substitution cost of phoneme pairs with the same final or initial consonant in the confusion matrix is significantly lower than the default value (for example, the confusion cost of the initial consonants of "huang" and "wang" in Minnan dialect is set to 0.2, much lower than the default substitution cost of 1.0). The phonetic similarity is defined as 1 minus the ratio of the weighted edit distance to the maximum of the pinyin lengths of the two sequences. The dialect phoneme confusion matrix is obtained through statistical analysis of speech recognition error samples of various dialects, covering three dimensions: initial consonant confusion, final consonant confusion, and tone confusion, which can effectively capture the pronunciation deviation patterns unique to dialects.

[0058] Weight coefficient 、 、 、 Joint learning is performed by minimizing the cross-entropy loss on the labeled dataset, satisfying the normalization constraint + + + = 1. In the experiments of this embodiment, the optimal weights learned are = 0.35, = 0.30, = 0.15, = 0.20. The candidate entity with the highest confidence score is selected as the final link result, that is, the entity e with the highest multi-signal fusion confidence score is selected from the candidate entity set . The structured trading instruction can be directly docked with the backend trading management system to achieve the full-process automation from voice input to order generation.

[0059] In summary, the core of this invention lies in: First, a multi-level progressive audio-visual fusion architecture is constructed. A bidirectional cross-attention mechanism with noise-aware adaptive temperature scaling is introduced at multiple levels of the encoder. Combining gated residual connections, deep interaction and fusion of audio features and visual features are achieved. And based on a differentiable generalized asymmetric Sigmoid mapping function, the modal weights are dynamically adjusted according to the real-time signal-to-noise ratio. When the signal-to-noise ratio is lower than 0 dB, the visual modal weight automatically increases to more than 70%; Second, we designed a cumulative multi-task elastic weight consolidation incremental learning framework for coastal dialects and a federated adaptive training mechanism based on dialect distribution similarity weighting. We protected the learned dialect knowledge through time decay regularization strategy and combined it with differential privacy protection with sensitivity calibration to achieve secure collaborative training among multiple markets. It supports adaptive recognition of eight coastal dialects, including Minnan, Cantonese, and Wu. Third, we construct a knowledge graph in the field of aquatic product trading and design an entity linking method with multi-signal fusion confidence scoring. We comprehensively score candidate entities based on four dimensions: semantic embedding similarity, contextual conditional probability, knowledge graph structural importance, and dialectal phonetic similarity, to achieve accurate understanding of transaction semantics and map vague, colloquial transaction expressions into structured transaction instructions.

[0060] This invention effectively solves the key technical bottlenecks of existing speech recognition technology, such as low recognition accuracy in noisy industrial environments, insufficient dialect support, and lack of semantic understanding in the industry. It provides reliable technical support for the intelligent upgrading of aquatic product trading markets and has broad prospects for industrial application and significant socio-economic value.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios, characterized in that, Includes the following steps: S1. Audio signal acquisition and preprocessing: Audio signals are acquired using a microphone array and noise reduction is performed to output the noise-reduced audio feature sequence. S2. Visual lip shape signal acquisition and feature extraction: Capture facial video sequences using a camera, extract the region of interest (ROI) of the lips, and use a 3D convolutional neural network to extract spatiotemporal features of the ROI to obtain a visual feature sequence. S3. Noise-aware multi-level progressive bimodal feature fusion: The audio feature sequence and the visual feature sequence are respectively input into the corresponding encoder network. A noise-aware adaptive temperature scaling bidirectional cross-attention mechanism is introduced at multiple preset levels of the encoder. The cross-attention output is adaptively integrated into the original features of the corresponding modality through gated residual connection to obtain the fused audio features and visual features. S4. Dynamic modal weight allocation based on signal-to-noise ratio: The signal-to-noise ratio of the current environment is estimated in real time, and after time-series smoothing, it is input into a differentiable generalized asymmetric Sigmoid mapping function to calculate the dynamic fusion weights of audio modality and visual modality. The fused audio features and visual features are weighted and summed based on the dynamic fusion weights to obtain the final multimodal fusion features; S5. Adaptive speech recognition for coastal dialects: Speech recognition is performed based on the multimodal fusion features to obtain text results; a cumulative multi-task elastic weight consolidation strategy is adopted, and a time decay regularization term for all learned dialect tasks is introduced into the loss function to achieve adaptive recognition of multiple coastal dialects. S6. Knowledge Graph-Based Transaction Semantic Understanding: Utilizing a pre-constructed knowledge graph in the aquatic product transaction domain, a multi-signal confidence scoring method integrating semantic embedding, contextual probability, graph structure importance, and dialect phonological similarity is employed to link entities in the identified text. After completing relation extraction and intent classification, the structured transaction instructions obtained through mapping are output.

2. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, The microphone array described in step S1 comprises 8 MEMS microphones; The noise reduction process is as follows: The acquired audio signal is input into a noise suppression module based on a deep neural network for noise reduction processing.

3. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, The camera mentioned in step S2 supports 120fps acquisition. The three-dimensional convolutional neural network consists of 5 three-dimensional convolutional layers and 3 fully connected layers. The kernel size of each three-dimensional convolutional layer is 3×3×3.

4. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, In step S3, audio features are used as key-value pairs between the query and visual features for cross-attention calculation, and visual features are used as key-value pairs between the query and audio features for cross-attention calculation. The specific implementation of the noise-aware adaptive temperature scaling bidirectional cross-attention mechanism and the gated residual connection is as follows: In each fusion layer of the encoder, the audio feature matrix is ​​transformed linearly to generate a query matrix, and the visual feature matrix is ​​transformed linearly to generate a key matrix and a value matrix, respectively. A noise-aware adaptive temperature coefficient is introduced, and the cross-attention output from audio to visual direction is calculated using a scaled dot product attention method. The attention score is obtained by multiplying the query matrix and the transpose of the key matrix, dividing by the product of the square root of the key vector dimension and the adaptive temperature coefficient, and then multiplying by the value matrix after softmax normalization. The adaptive temperature coefficient is generated by mapping the signal-to-noise ratio estimated in the current frame through a ReLU activation function after linear transformation, ensuring that the temperature coefficient is not lower than 1. The weights and biases of the linear transformation are independent learnable parameters for each fusion layer. The cross-attention output from the visual to the audio direction is calculated in the same way, and its corresponding adaptive temperature coefficient is independently parameterized. The cross-attention outputs from each direction are integrated into the original features of the corresponding modality through a gated residual connection mechanism: After concatenating the original audio features and the cross-attention output along the feature dimension, the element-wise gating factor is generated by linear transformation and Sigmoid activation function. The cross-attention output is then gated and modulated by element-wise multiplication and added to the original features by residual addition. Finally, the updated audio features are output after layer normalization. The visual features are updated using the same gated residual method.

5. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, The specific implementation method of dynamic modal weight allocation in step S4 is as follows: The signal-to-noise ratio (SNR) of the current audio frame is estimated in real time using the spectrum analysis module, resulting in the instantaneous SNR. The instantaneous signal-to-noise ratio is then subjected to exponential moving average time-series smoothing. ； in, For smoothing coefficients, For the first Signal-to-noise ratio after smoothing. for The smooth signal-to-noise ratio at any given moment; Smoothed signal-to-noise ratio Calculate the visual modality weights by taking a differentiable generalized asymmetric sigmoid mapping function as input: ； in, A learnable temperature parameter for controlling the steepness of the weight transition. To control the learnable offset parameter of the weight transition center point, Learnable shape parameters for controlling the asymmetry of the weighted transition curve; Audio modal weights ; The dynamic fusion weights are applied to the weighted summation of the fused features to obtain the final multimodal fusion features: ； in, The audio features are those obtained after multi-level cross-attention fusion. These are visual features resulting from multi-level cross-attention fusion.

6. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, The specific implementation method of the cumulative multi-task elastic weight consolidation strategy in step S5 is as follows: When the first is introduced in sequence When fine-tuning the model using dialect data, a method specific to all dialects is introduced into the loss function. Cumulative regularization terms for a learned dialect task: ； in, For the current number The loss in dialect recognition task, For the first The first task corresponding to the learned dialect Fisher information content of each parameter For the model number The current values ​​of the parameters, To complete the first The parameter values ​​saved after learning a dialect task; Regularization weights for each learned task A time decay strategy is adopted for allocation, which is calculated by multiplying the basic regularization coefficient by a negative exponential decay function with respect to the temporal distance of the task, so that the dialects learned recently are better protected by knowledge, and the decay rate is an adjustable hyperparameter.

7. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, Step S5 also includes a federated adaptive training mechanism based on dialect distribution similarity weighting, which is implemented as follows: Terminal devices deployed in multiple aquatic product trading markets each use local dialect speech data for model training, and only the model parameter updates are uploaded to the central aggregation server, while the original data does not leave the domain; Differential privacy protection with sensitivity calibration is applied before parameter upload, which means adding a Gaussian noise vector to the local model parameters of each market. The vector has a mean of zero, a variance proportional to the product of the square of the privacy noise coefficient and the square of the gradient clipping threshold, and is inversely proportional to the amount of local data. The central aggregation server uses a federated average algorithm based on dialect distribution similarity weighting to aggregate model parameters. The aggregation weight of each market is obtained by normalizing the product of its local data volume and dialect distribution similarity weight. The dialect distribution similarity weight is defined as the negative exponent of the mean Jensen-Shannon divergence of dialect phoneme distribution among markets.

8. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, The specific implementation method of the multi-signal fusion confidence scoring method in step S6 is as follows: For each text segment detected in the identified text Retrieve a set of candidate entities from the aquatic product trading knowledge graph, and for each candidate entity... Calculate the confidence score for multi-signal fusion: ； in, The cosine similarity between the candidate entity embedding vector and the mentioned text embedding vector is... For context-based windows Entity conditional probability, Assess the importance of entities within the knowledge graph. For phonological similarity scoring based on dialect phoneme confusion matrix, , , , Learnable weight coefficients to satisfy normalization constraints; The candidate entity with the highest confidence score is selected as the link result. Based on the linked entities, relationship extraction and intent classification are completed to generate a structured trading instruction containing elements of variety, specification, quality, and price.

9. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, The knowledge graph in the aquatic product trading field described in step S6 is constructed by combining top-down ontology design with bottom-up instance filling. The ontology layer defines four core entity types: variety entity, specification entity, quality entity, and price entity, and defines the synonym relationships, hierarchical relationships, attribute association relationships, and co-occurrence relationships between entities.

10. The dual-modal adaptive speech recognition method for high-noise aquatic product trading scenarios according to claim 1, characterized in that, Step S5 also includes a speaker adaptation step: When a new user uses the app for the first time, the app reads aloud preset commonly used trading phrases. Based on the collected samples, the model's attention layer is fine-tuned using a low-rank decomposition strategy, and the fine-tuned personalized parameters are stored on the user's local device.

Images