A crosswise intermodal transport obfuscation method and apparatus based on adaptive optimal transport

Abstract

This invention relates to the field of speech translation technology, and particularly to a cross-transport obfuscation method and apparatus based on adaptive optimal transmission. The method includes: constructing an optimal transmission model within a multi-task general framework; performing attention-enhanced optimal transmission alignment on speech and text sequences; optimizing the attention-enhanced optimal transmission alignment based on a dynamic window strategy to obtain the alignment relationship between the speech and text sequences; fusing speech and text features of the speech and text sequences through a similarity-based adaptive fusion strategy; and combining a contrastive learning loss function with a multi-task learning framework, constructing positive and negative sample pairs, and combining the multi-task loss functions to obtain a unified optimization objective. This invention effectively reduces the representational differences between speech and text through dynamic window strategies, optimal transmission, and contrastive learning, achieving significant improvements in translation tasks for low-resource languages.

Description

Technical Field

[0001] This invention relates to the field of speech translation technology, and in particular to a cross-transport obfuscation method and apparatus based on adaptive optimal transmission. Background Technology

[0002] Speech-to-Text Translation (S2TT) is an important research area in artificial intelligence, with the core goal of directly converting speech signals from one language into text in another. Traditional methods typically employ a cascaded framework, first converting speech into source language text using Automatic Speech Recognition (ASR), and then translating the text into the target language using Machine Translation (MT).

[0003] Although this cascaded approach is widely used in industry, it has two significant bottlenecks. First, it causes error propagation, where recognition errors in the ASR stage propagate directly to the MT stage, resulting in a gradual decline in translation quality. Second, it has the problem of modal differences, as speech and text are heterogeneous modalities, and their deep semantic relationships are difficult to model effectively during the cascaded process.

[0004] To overcome these limitations, end-to-end speech translation (E2E-ST) technology has emerged. Unlike cascaded frameworks, E2E-ST models directly map speech signals to target language text, skipping the intermediate transcription process. Theoretically, this can reduce error propagation and enhance cross-modal association learning.

[0005] In recent years, a number of representative studies have emerged in the field of end-to-end speech translation. Bérard first proposed an end-to-end ST model based on an attention mechanism, which significantly improved translation efficiency by dynamically aligning speech frames with text tags. He further introduced pre-trained speech encoders and text decoders to enhance the model's generalization ability using large-scale unsupervised data. He also proposed a multimodal joint training strategy to reduce the difference between speech and text representations by sharing the parameter space.

[0006] Despite significant progress in model architecture and training strategies, the cross-modal representation gap remains fundamentally unresolved. Specifically, speech, as a continuous time-varying signal, has a high-density representation space with strong local correlations, while text, as a discrete sequence of symbols, has a sparse representation space with explosive combinatorial properties. This fundamental difference leads to a significant semantic gap in intermodal mapping.

[0007] To address the modal gap issue, existing research mainly explores it from two dimensions: cross-modal alignment and representation enhancement.

[0008] In cross-modal alignment, mainstream methods typically rely on external alignment tools or fixed hybrid strategies. For example, word-level hybrid strategies require a forced aligner to predefine the mapping relationship between speech and text, a process that necessitates large amounts of parallel corpora or manually annotated data. However, in low-resource languages, acquiring such data is extremely costly or even infeasible, making it difficult for models to capture the semantic connections between speech and text, and even causing serious alignment biases due to annotation noise.

[0009] Some studies also employ a hybrid strategy to directly fuse speech and text features, but the mixing ratio is fixed and still relies on external alignment tools, limiting flexibility in practical applications. When the correspondence between speech frames and text tags changes dynamically, static hybrid strategies cannot adaptively adjust the weights, leading to semantic gaps in the translation results.

[0010] While optimal transport (OT) methods, such as CMOT, achieve global alignment by modeling speech and text as probability distributions, and utilize optimal transport to find alignment relationships between speech and text sequences and mix different modalities at the tag level, they still fall short when dealing with complex linguistic phenomena. Furthermore, traditional OT frameworks lack the ability to dynamically adapt to changes in language characteristics and context, causing alignment results to deviate from the true semantics in complex scenarios.

[0011] In terms of representation enhancement, the cross-modal contrastive learning method ConST improves feature consistency by constructing positive and negative speech-text pairs, significantly increasing cross-modal retrieval accuracy. However, ConST still suffers from an imprecise negative sample selection mechanism. Its negative sample selection relies on a random sampling strategy, which may mistakenly classify semantically related samples with different alignment positions as negative pairs, failing to fully utilize the internal structural information of the modality for refined contrastive learning.

[0012] More importantly, existing methods fail to effectively integrate contrastive learning, OT alignment, and hybrid strategies, resulting in fragmented optimization: the contrastive learning objective does not utilize the alignment relationships generated by OT to select high-quality samples, and the hybrid strategy does not introduce contrastive constraints to enhance the semantic consistency of feature fusion. This isolated optimization limits the model's generalization ability in low-resource scenarios, making it difficult to break through the performance ceiling of cross-modal translation.

[0013] The main shortcomings of existing speech-to-text translation technologies are as follows:

[0014] 1. Low cross-modal alignment accuracy. Speech and text differ significantly in features, structure, and grammar, resulting in a modal gap between their representation spaces. This is the main bottleneck affecting the effectiveness of end-to-end speech translation.

[0015] 2. Insufficient adaptability of cross-modal alignment. Existing cross-modal alignment methods usually rely on external alignment tools or fixed alignment strategies, lacking the ability to adapt to different languages ​​and contexts, and are difficult to effectively handle situations where there is a mismatch between speech and text lengths or significant changes in speech rate.

[0016] 3. Lack of flexibility in hybridization strategies. Existing hybridization methods mostly use fixed mixing ratios, which cannot be dynamically adjusted according to the similarity between speech and text features, thus limiting the natural transition and semantic coherence of cross-modal representations.

[0017] 4. Difficulties in low-resource language translation. In resource-constrained language environments, traditional methods struggle to obtain sufficient training data and lack effective mechanisms for multimodal learning and cross-modal knowledge transfer, leading to a significant decline in translation performance. Summary of the Invention

[0018] To address the technical problems of low cross-modal alignment accuracy, insufficient adaptability of cross-modal alignment, lack of flexibility in hybrid strategies, and difficulties in translating low-resource languages ​​in existing technologies, this invention provides a cross-modal transport obfuscation method and apparatus based on adaptive optimal transmission. The technical solution is as follows:

[0019] On the one hand, a cross-modal transport obfuscation method based on adaptive optimal transmission is provided, characterized in that the method includes:

[0020] S1. Construct the optimal transmission model for a multi-task general framework; wherein, the optimal transmission model includes a speech encoder, a word embedding layer, a Transformer encoder, an optimal alignment module, an adaptive mixing module, and a Transformer decoder; the word embedding layer is set up in parallel with the speech encoder;

[0021] S2. Token-level feature representations of the speech sequence and text sequence are obtained through the speech encoder and word embedding layer, respectively; the Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module;

[0022] S3. Input the encoded features output by the Transformer encoder into the optimal alignment module, optimize the attention enhancement alignment through the dynamic window strategy, perform optimal transmission calculation using the OT calculation submodule, and generate a cross-modal alignment matrix.

[0023] S4. Input the cross-modal alignment matrix and encoded features into the adaptive fusion module. Based on the similarity-based adaptive fusion strategy, fuse the speech and text features of the speech sequence and text sequence to obtain a cross-modal representation with natural transition and semantic coherence.

[0024] S5. Input the cross-modal hybrid feature representation into the Transformer decoder, and perform joint training by combining the contrastive learning loss function and the multi-task learning framework to achieve end-to-end cross-modal feature confusion optimization.

[0025] Optionally, S1 also includes:

[0026] The speech encoder is responsible for processing speech input and generating a feature representation of the speech;

[0027] The word embedding layer converts text input into an embedded representation;

[0028] The Transformer encoder further processes the speech and text representations, and finally the Transformer decoder generates the target translation output.

[0029] Optionally, token-level feature representations of the speech sequence and text sequence are obtained through a speech encoder and a word embedding layer, respectively; the Transformer encoder encodes the speech encoder output and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module, including:

[0030] By treating speech sequences and text sequences as two independent probability distributions, the optimal transmission problem is formally defined.

[0031] The cost between speech and text features is calculated using Euclidean distance as a preliminary basis for alignment.

[0032] L2 norm normalization is performed on speech and text features respectively to provide standardized input for attention calculation;

[0033] By utilizing the normalized features, attention weight matrices within speech and text sequences are calculated using a self-attention mechanism to capture the dependencies within the sequences.

[0034] By combining the self-attention weights with the original distance matrix, an attention-enhanced distance metric matrix is ​​generated.

[0035] The initial alignment relationship between speech and text tokens is obtained from the enhanced distance matrix by using the minimum distance principle, which provides a foundation for subsequent precise alignment based on the optimal transmission algorithm.

[0036] Optionally, a preliminary alignment relationship between speech and text tokens is obtained from the enhanced distance matrix using the minimum distance principle, providing a foundation for subsequent precise alignment based on the optimal transmission algorithm, including:

[0037] The speech and text token feature vectors that have obtained the initial alignment relationship are respectively constructed as source probability distribution and target probability distribution;

[0038] The distance values ​​in the enhanced distance matrix are used as the transmission cost function between the voice token and the text token.

[0039] The optimal transmission algorithm with relaxed constraints is adopted, which retains only row constraints and removes the strict requirements of column constraints;

[0040] Solve for the optimal transmission matrix under relaxed constraints to obtain the optimal transmission volume allocation from voice token to text token;

[0041] The optimal text token position for each voice token is determined based on the optimal transmission matrix, generating a precise token-level alignment relationship.

[0042] Optionally, the encoded features output by the Transformer encoder are input into the optimal alignment module, attention-enhanced alignment is optimized using a dynamic window strategy, and optimal transfer calculation is performed using the OT computation submodule to generate a cross-modal alignment matrix, including:

[0043] The base window size is calculated based on the length characteristics of the speech and text sequences and adjusted using a scaling factor.

[0044] The window size is dynamically adjusted based on the degree of deviation in the sequence length ratio;

[0045] Based on the position of the source sequence, calculate the window range of the target sequence to ensure that the window can adapt to changes in the length of the speech and text sequences;

[0046] Determine the valid positions based on the padding mask, convert the window mask into binary form, and filter valid alignment candidates.

[0047] The mask is applied to the distance matrix after attention enhancement to maximize the distance to the location outside the window; the optimal target location corresponding to each source location is calculated to obtain the alignment relationship between the speech sequence and the text sequence.

[0048] Optionally, the cross-modal alignment matrix and encoded features are input into the adaptive fusion module, and the speech and text features of the speech sequence and text sequence are fused based on a similarity-based adaptive fusion strategy to obtain a cross-modal representation with natural transitions and semantic coherence, including:

[0049] Calculate the cosine similarity between speech features and text features, and use it as the basis for the mixed weights;

[0050] use The function performs a smooth transformation on the similarity values ​​to obtain the weights used for mixing;

[0051] The mixing coefficient is dynamically adjusted based on similarity to control the fusion ratio of speech and text features;

[0052] The calculated mixing coefficients are used to smoothly fuse speech and text features to obtain cross-modal representations with natural transitions and semantic coherence.

[0053] Optionally, the cross-modal hybrid feature representation is input into the Transformer decoder, and joint training is performed using a contrastive learning loss function and a multi-task learning framework to achieve end-to-end cross-modal feature obfuscation optimization, including:

[0054] High-quality positive and negative sample pairs are constructed by optimizing transmission alignment. Positive sample pairs consist of aligned speech and text features, while negative sample pairs consist of misaligned features.

[0055] The InfoNCE loss function is used to bring positive sample pairs closer together while pushing negative sample pairs further apart.

[0056] By combining contrastive learning loss with multi-task loss functions such as speech translation loss and machine translation loss, a unified optimization objective is formed to achieve end-to-end cross-modal feature confusion optimization.

[0057] On the other hand, a straddle-transport obfuscation device based on adaptive optimal transmission is provided. This device is applied to a straddle-transport obfuscation method based on adaptive optimal transmission. The device includes:

[0058] The model building module is used to construct the optimal transmission model for a multi-task general framework. The optimal transmission model includes a speech encoder, a word embedding layer, a Transformer encoder, an optimal alignment module, an adaptive mixing module, and a Transformer decoder. The word embedding layer is set up in parallel with the speech encoder.

[0059] The feature encoding module is used to obtain token-level feature representations of speech sequences and text sequences through the speech encoder and word embedding layer, respectively; the Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module;

[0060] The cross-modal alignment module is used to input the encoded features output by the Transformer encoder into the optimal alignment module, optimize the attention enhancement alignment through a dynamic window strategy, perform optimal transfer calculation using the OT calculation submodule, and generate a cross-modal alignment matrix.

[0061] The cross-modal representation module is used to input the cross-modal alignment matrix and encoded features into the adaptive fusion module. Based on the similarity-based adaptive fusion strategy, the speech and text features of the speech sequence and text sequence are fused to obtain a cross-modal representation with natural transition and semantic coherence.

[0062] The optimization module is used to input cross-modal hybrid feature representations into the Transformer decoder, and perform joint training by combining the contrastive learning loss function and the multi-task learning framework to achieve end-to-end cross-modal feature confusion optimization.

[0063] On the other hand, a straddle-transport obfuscation device based on adaptive optimal transmission is provided, the straddle-transport obfuscation device based on adaptive optimal transmission includes: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for straddle-transport obfuscation based on adaptive optimal transmission.

[0064] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of the straddle-transport obfuscation method based on adaptive optimal transmission.

[0065] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0066] In this embodiment of the invention, a cross-modal alignment method based on adaptive optimal transmission is proposed. The method performs adaptive weight confusion on the speech and text features after optimal transmission alignment to generate a modal fusion feature representation. Through an innovative attention enhancement mechanism and dynamic window strategy, the alignment accuracy between speech and text is significantly improved, especially for low-resource languages, effectively solving the translation error caused by modal differences.

[0067] This invention innovatively introduces an adaptive fusion strategy based on feature similarity, which dynamically adjusts the fusion weights of speech and text features, making cross-modal representations more natural and continuous. In particular, it demonstrates stronger robustness and adaptability when there is a large difference in the length of speech and text.

[0068] The contrastive learning loss function proposed in this invention, combined with a multi-task learning framework, effectively addresses the data scarcity problem in speech translation tasks, particularly enhancing the system's generalization ability in low-resource environments. The contrastive learning module utilizes the OT alignment matrix to construct positive and negative sample pairs and employs InfoNCE loss to narrow the distance between aligned features and widen the distance between unaligned features, thereby enhancing cross-modal representation learning capabilities. Simultaneously, the integration with a multi-task learning framework (including speech translation, machine translation, and cross-modal hybrid tasks) enables the model to fully leverage more data sources for learning, improving its generalization ability and reducing its dependence on large amounts of labeled data. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A flowchart illustrating a cross-modal transport obfuscation method based on adaptive optimal transmission, provided in an embodiment of the present invention;

[0071] Figure 2 This is a general framework diagram of the model provided in the embodiments of the present invention;

[0072] Figure 3 A block diagram of a cross-transport obfuscation device based on adaptive optimal transmission provided in an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of the overall architecture of the model provided in the embodiments of the present invention. Detailed Implementation

[0074] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0075] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0076] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0077] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0078] This invention provides a straddle-based intermodal transport obfuscation method based on adaptive optimal transmission. This method can be implemented by a straddle-based intermodal transport obfuscation device, which can be a terminal or a server. Figure 1 The flowchart of the straddle-based obfuscation method based on adaptive optimal transmission is shown below. Figure 1 As shown, the cross-transport obfuscation method based on adaptive optimal transmission proposed in this invention, such as... Figure 2The overall framework of the model provided by this invention; the processing flow of this method may include the following steps:

[0079] S1. Construct the optimal transmission model for a multi-task general framework; wherein, the optimal transmission model includes a speech encoder, a word embedding layer, a Transformer encoder, an optimal alignment module, an adaptive mixing module, and a Transformer decoder; the word embedding layer is set up in parallel with the speech encoder.

[0080] In one feasible implementation, the model architecture is as follows: Figure 2 This method effectively reduces the representational differences between speech and text through techniques such as dynamic windowing strategy, optimal transmission, and contrastive learning, and achieves significant improvements in translation tasks for low-resource languages.

[0081] In one possible implementation, S1 further includes:

[0082] The speech encoder is responsible for processing speech input and generating a feature representation of the speech;

[0083] The word embedding layer converts text input into an embedded representation;

[0084] The Transformer encoder further processes the speech and text representations, and finally the Transformer decoder generates the target translation output.

[0085] S2. Token-level feature representations of the speech sequence and text sequence are obtained through the speech encoder and word embedding layer, respectively. The Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module.

[0086] In one feasible implementation, token-level feature representations of the speech sequence and text sequence are obtained through a speech encoder and a word embedding layer, respectively; the Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module, including:

[0087] By treating speech sequences and text sequences as two independent probability distributions, the optimal transmission problem is formally defined.

[0088] The cost between speech and text features is calculated using Euclidean distance as a preliminary basis for alignment.

[0089] L2 norm normalization is performed on speech and text features respectively to provide standardized input for attention calculation;

[0090] By utilizing the normalized features, attention weight matrices within speech and text sequences are calculated using a self-attention mechanism to capture the dependencies within the sequences.

[0091] By combining the self-attention weights with the original distance matrix, an attention-enhanced distance metric matrix is ​​generated.

[0092] The initial alignment relationship between speech and text tokens is obtained from the enhanced distance matrix by using the minimum distance principle, which provides a foundation for subsequent precise alignment based on the optimal transmission algorithm.

[0093] In one feasible implementation, a preliminary alignment relationship between speech and text tokens is obtained from the enhanced distance matrix using the minimum distance principle, providing a foundation for subsequent precise alignment based on the optimal transmission algorithm, including:

[0094] The speech and text token feature vectors that have obtained the initial alignment relationship are respectively constructed as source probability distribution and target probability distribution;

[0095] The distance values ​​in the enhanced distance matrix are used as the transmission cost function between the voice token and the text token.

[0096] The optimal transmission algorithm with relaxed constraints is adopted, which retains only row constraints and removes the strict requirements of column constraints;

[0097] Solve for the optimal transmission matrix under relaxed constraints to obtain the optimal transmission volume allocation from voice token to text token;

[0098] The optimal text token position for each voice token is determined based on the optimal transmission matrix, generating a precise token-level alignment relationship.

[0099] In one feasible implementation, the encoded features output by the Transformer encoder are input into the optimal alignment module, attention-enhanced alignment is optimized using a dynamic window strategy, and optimal transfer calculation is performed using the OT computation submodule to generate a cross-modal alignment matrix, including:

[0100] The base window size is calculated based on the length characteristics of the speech and text sequences and adjusted using a scaling factor.

[0101] The window size is dynamically adjusted based on the degree of deviation in the sequence length ratio;

[0102] Based on the position of the source sequence, calculate the window range of the target sequence to ensure that the window can adapt to changes in the length of the speech and text sequences;

[0103] Determine the valid positions based on the padding mask, convert the window mask into binary form, and filter valid alignment candidates.

[0104] The mask is applied to the distance matrix after attention enhancement to maximize the distance to the location outside the window; the optimal target location corresponding to each source location is calculated to obtain the alignment relationship between the speech sequence and the text sequence.

[0105] In one feasible implementation, the speech sequence and text sequence are treated as two independent probability distributions, and the optimal transmission problem is formally defined. To improve computational efficiency, a relaxed definition of optimal transmission is adopted:

[0106]

[0107]

[0108] in, Indicates the optimal transmission distance. Represents a speech sequence. Represents a text sequence. It is a cost function. i Indicates the first word in the speech sequence i Index of each token j Indicates the first in the text sequence j Index of each token Let the optimal transmission matrix represent the amount of data transmitted from voice token i to text token j. This refers to the weight of the voice token. The traditional optimal transmission problem requires both strict row and column constraints to be satisfied. However, by relaxing the constraints, only the row constraints are retained, and the strict requirements of the column constraints are removed, thus simplifying the computational complexity while retaining enough information to determine the cross-modal alignment relationship.

[0109] By relaxing the strict constraints of traditional optimal transmission and adopting a dynamic window strategy to limit the alignment search range, the computational complexity is significantly reduced, while retaining enough information to determine cross-modal alignment relationships.

[0110] In one feasible implementation, calculating the original distance matrix and feature normalization includes:

[0111] The distance between speech and text features is calculated using Euclidean distance as the initial cost of alignment.

[0112]

[0113] To enhance stability, speech and text features are normalized to eliminate differences in magnitude between different features.

[0114]

[0115] in, This represents the normalized speech token feature vector. This represents the normalized text token feature vector.

[0116] Self-attention weight calculation includes:

[0117] Self-attention mechanisms are used to calculate self-attention weights within speech and text sequences, capturing dependencies within the sequences and improving the semantic consistency of alignment. The calculation formula is as follows:

[0118]

[0119] in, Represents the normalized attention weight matrix of the speech sequence. The normalized attention weight matrix represents the text sequence; The original alignment score matrix representing the speech sequence; This represents the original alignment score matrix of the text sequence.

[0120] Enhanced distance matrix and optimal alignment include:

[0121] By combining the original distance matrix with the attention-enhanced distance matrix, an enhanced distance metric is generated to ensure high-quality alignment. The final alignment is obtained from the enhanced matrix using the minimum distance principle.

[0122]

[0123] in, This represents the enhanced distance matrix; Represents the original distance matrix; The weighting coefficients representing attention enhancement are used to control the degree to which the attention mechanism affects the distance matrix.

[0124] The final alignment matrix is:

[0125]

[0126] in, Voice token The corresponding optimal text position index.

[0127] The above steps enable the optimal transport alignment mechanism for attention enhancement, achieving precise alignment between speech and text sequences and laying the foundation for subsequent cross-modal mixing.

[0128] In this embodiment of the invention, an optimal transmission alignment mechanism based on attention enhancement is constructed. By embedding a self-attention mechanism into the optimal transmission theory framework, the internal semantic weights of the speech and text sequences are dynamically calculated, eliminating dependence on external alignment tools. Specifically, by utilizing a self-attention matrix to capture the temporal dependencies between speech frames and the grammatical structural constraints of the text, alignment accuracy can be significantly improved.

[0129] S3. Input the encoded features output by the Transformer encoder into the optimal alignment module, optimize the attention enhancement alignment through the dynamic window strategy, perform optimal transmission calculation using the OT calculation submodule, and generate a cross-modal alignment matrix.

[0130] In one feasible implementation, the dynamic window strategy further optimizes the alignment search process based on the optimal transmission of attention enhancement, thereby improving alignment accuracy and efficiency. This strategy works synergistically with the attention enhancement mechanism to jointly improve cross-modal alignment quality.

[0131] In one feasible implementation, the encoded features output by the Transformer encoder are input into the optimal alignment module, attention-enhanced alignment is optimized using a dynamic window strategy, and optimal transfer calculation is performed using the OT computation submodule to generate a cross-modal alignment matrix, including:

[0132] The base window size is calculated based on the length characteristics of the speech and text sequences and adjusted using a scaling factor.

[0133] The window size is dynamically adjusted based on the degree of deviation in the sequence length ratio;

[0134] Based on the position of the source sequence, calculate the window range of the target sequence to ensure that the window can adapt to changes in the length of the speech and text sequences;

[0135] Determine the valid positions based on the padding mask, convert the window mask into binary form, and filter valid alignment candidates.

[0136] The mask is applied to the distance matrix after attention enhancement to maximize the distance to the location outside the window; the optimal target location corresponding to each source location is calculated to obtain the alignment relationship between the speech sequence and the text sequence.

[0137] In one feasible implementation, the calculation of the base window size includes:

[0138] The base window size is calculated based on the length characteristics of speech and text, and adjusted using a scaling factor to accommodate sequences of different lengths.

[0139]

[0140] Adaptive window adjustment includes:

[0141] The window size is dynamically adjusted based on the degree of deviation in the sequence length ratio.

[0142]

[0143]

[0144] in, Indicates the first i The length of the source sequence for each sample; Indicates the first i The target sequence length for each sample; base_window represents the initial window size; adaptive_window represents the adaptive window size.

[0145] When the actual ratio deviates from the ideal value, the window size will increase accordingly to accommodate possible non-linear alignment relationships. This design is particularly suitable for cross-modal sequences such as speech and text, which naturally exhibit speed variations.

[0146] Determining the window extent and alignment includes:

[0147] Based on the position of the source sequence, the window range of the target sequence is calculated to ensure that the window can adapt to changes in the length of speech and text, maximizing alignment accuracy.

[0148] For each source sequence position Calculate its mapping center point in the target sequence:

[0149]

[0150] To ensure boundary security, center point constraints are applied:

[0151]

[0152] in, It is the length of the target sequence.

[0153] Determine the window search range based on the center point and the adaptive window size:

[0154]

[0155]

[0156] Masking and final alignment calculations include:

[0157] Apply a padding mask to ensure that only valid positions are considered:

[0158]

[0159] Convert the window mask to binary form for filtering valid alignment candidates:

[0160]

[0161] Applying a mask to the attention-enhanced distance matrix maximizes the distance to positions outside the window:

[0162]

[0163] in, Usually, a very large value is chosen (e.g., 6 × 10). 4 ).

[0164] Finally, the optimal target location for each source location is calculated:

[0165]

[0166] In one feasible implementation, the dynamic window strategy significantly improves the accuracy and robustness of cross-modal alignment, especially when dealing with sequences with length mismatches or significant changes in speech rate. By organically combining sequence length characteristics, positional mapping relationships, and feature similarity, this method can adapt to various complex alignment scenarios, providing a reliable foundation for subsequent cross-modal mixing.

[0167] S4. Input the cross-modal alignment matrix and encoded features into the adaptive fusion module. Based on the similarity-based adaptive fusion strategy, fuse the speech and text features of the speech sequence and text sequence to obtain a cross-modal representation with natural transition and semantic coherence.

[0168] In one feasible implementation, after obtaining the alignment relationship between speech and text, the present invention fuses speech and text features through an adaptive fusion strategy to ensure a natural transition and semantic coherence in cross-modal representation.

[0169] In one feasible implementation, the cross-modal alignment matrix and encoded features are input into the adaptive fusion module. Based on a similarity-based adaptive fusion strategy, the speech and text features of the speech sequence and text sequence are fused to obtain a cross-modal representation with natural transitions and semantic coherence, including:

[0170] Calculate the cosine similarity between speech features and text features, and use it as the basis for the mixed weights;

[0171] use The function performs a smooth transformation on the similarity values ​​to obtain the weights used for mixing;

[0172] The mixing coefficient is dynamically adjusted based on similarity to control the fusion ratio of speech and text features;

[0173] The calculated mixing coefficients are used to smoothly fuse speech and text features to obtain cross-modal representations with natural transitions and semantic coherence.

[0174] In one feasible implementation, the cosine similarity between speech features and text features is calculated as the basis for the mixing weights:

[0175]

[0176] use The function performs a smooth transformation on the similarity values ​​to obtain the weights used for blending:

[0177]

[0178] The mixing coefficient is dynamically adjusted based on similarity to control the fusion ratio of speech and text features:

[0179]

[0180] in, This represents the basic mixing probability, controlling the sensitivity of adaptive adjustment. The formula achieves adaptive fusion through linear interpolation: when sim_weight = 0 (low similarity). The mixing coefficient tends to be a fixed value; when sim_weight = 1 (high similarity). It completely preserves speech features; when sim_weight varies between 0 and 1, exist The dynamic adjustment between 1 and 2. Parameters control the adjustment range: The larger the similarity, the more significant the impact of the similarity on the mixing ratio; The smaller the value, the more stable the mixing ratio. Through this mechanism, the system automatically finds the optimal balance between preserving speech features and fusing textual information based on the alignment quality.

[0181] By utilizing the calculated mixing coefficients to smoothly fuse speech and text features, the abrupt problem of binary selection is avoided, achieving a natural transition between speech and text features and obtaining a cross-modal representation with natural transition and semantic coherence.

[0182]

[0183] Through the above steps, the similarity-based adaptive fusion strategy achieves intelligent fusion of speech and text features. This invention dynamically adjusts the fusion coefficients according to the feature cosine similarity to achieve continuous and smooth cross-modal feature fusion. Without this strategy, the fusion result would be unnatural.

[0184] S5. Input the cross-modal hybrid feature representation into the Transformer decoder, and perform joint training by combining the contrastive learning loss function and the multi-task learning framework to achieve end-to-end cross-modal feature confusion optimization.

[0185] In one feasible implementation, the present invention combines a contrastive learning loss function with a multi-task learning framework to enhance the learning ability of cross-modal representations by constructing high-quality positive and negative sample pairs.

[0186] In one feasible implementation, the cross-modal hybrid feature representation is input into the Transformer decoder, and joint training is performed using a contrastive learning loss function and a multi-task learning framework to achieve end-to-end cross-modal feature obfuscation optimization, including:

[0187] High-quality positive and negative sample pairs are constructed by optimizing transmission alignment. Positive sample pairs consist of aligned speech and text features, while negative sample pairs consist of misaligned features.

[0188] The InfoNCE loss function is used to shorten the distance between positive sample pairs while widening the distance between negative sample pairs, thereby enhancing the matching degree between speech and text features.

[0189]

[0190] in, This represents the contrastive learning loss function. Indicates the first i The set of voice tokens in each sample This indicates the specific voice token. It's a temperature parameter that controls the sharpness of the distribution. It refers to the batch size. It is a sample A set of effective speech feature indexes;

[0191] By combining contrastive learning loss with multi-task loss functions such as speech translation loss and machine translation loss, a unified optimization objective is formed, improving the model's generalization ability in multimodal tasks. The final total loss function is:

[0192]

[0193] The loss in speech translation is:

[0194]

[0195] Machine translation loss:

[0196]

[0197] Mixed sequence translation loss:

[0198]

[0199] in, It is a sequence obtained through optimal transmission alignment mixing.

[0200] KL divergence loss:

[0201]

[0202]

[0203] in, , and These represent the predicted distributions for the mixed sequence, speech sequence, and text sequence, respectively. , and It is a weight hyperparameter used to balance the contributions of each loss term.

[0204] In one feasible implementation, the coordinated adjustment of these parameters ensures the balance between contrastive learning and other learning objectives, forming an efficient multi-task learning framework. High-quality positive and negative sample pairs are constructed based on the OT alignment results through the contrastive learning loss function, the InfoNCE loss is used to shorten the distance between aligned features, and multi-task joint training is combined with speech translation loss, machine translation and KL divergence loss to optimize cross-modal representation and improve the system's generalization ability in low-resource environments. Without this framework, it would be impossible to effectively utilize limited data to construct the mapping relationship between speech and text features.

[0205] In this embodiment of the invention, a cross-modal alignment method based on adaptive optimal transmission is first proposed. Through an innovative attention enhancement mechanism and dynamic window strategy, the alignment accuracy between speech and text is significantly improved, especially for low-resource languages, effectively solving the translation error caused by modal differences.

[0206] Secondly, this invention innovatively introduces an adaptive fusion strategy based on feature similarity, which dynamically adjusts the fusion weights of speech and text features, making cross-modal representation more natural and continuous. In particular, it demonstrates stronger robustness and adaptability when the speech and text lengths differ significantly.

[0207] Finally, the contrastive learning loss function proposed in this invention, combined with a multi-task learning framework, effectively addresses the data scarcity problem in speech translation tasks, particularly enhancing the system's generalization ability in low-resource environments. Through contrastive learning, positive and negative sample pairs are constructed between speech and text features. The InfoNCE loss method narrows the distance between aligned features and widens the distance between unaligned features, thereby enhancing cross-modal representation learning capabilities. Simultaneously, the integration of a multi-task learning framework (including speech translation, machine translation, and cross-modal hybrid tasks) allows the model to fully utilize more data sources for learning, improving its generalization ability and reducing its dependence on large amounts of labeled data.

[0208] Figure 3 This is a block diagram of a straddle-based intermodal obfuscation apparatus 300 based on adaptive optimal transmission, according to an exemplary embodiment. The apparatus 300 is used in a straddle-based intermodal obfuscation method based on adaptive optimal transmission. (Refer to...) Figure 3 The device includes a model building module 310, a feature encoding module 320, a cross-modal alignment module 330, a cross-modal representation module 340, and an optimization module 350. Wherein:

[0209] The model building module 310 is used to build the optimal transmission model of the multi-task general framework; wherein, the optimal transmission model includes a speech encoder, a word embedding layer, a Transformer encoder, an optimal alignment module, an adaptive mixing module, and a Transformer decoder; the word embedding layer is set in parallel with the speech encoder;

[0210] The feature encoding module 320 is used to obtain token-level feature representations of speech sequences and text sequences through the speech encoder and word embedding layer, respectively; the Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module;

[0211] The cross-modal alignment module 330 is used to input the encoded features output by the Transformer encoder into the optimal alignment module, optimize the attention enhancement alignment through a dynamic window strategy, perform optimal transmission calculation using the OT calculation submodule, and generate a cross-modal alignment matrix.

[0212] The cross-modal representation module 340 is used to input the cross-modal alignment matrix and encoded features into the adaptive fusion module, and fuse the speech and text features of the speech sequence and text sequence based on the similarity-based adaptive fusion strategy 350 to obtain a cross-modal representation with natural transition and semantic coherence.

[0213] The optimization module is used to input cross-modal hybrid feature representations into the Transformer decoder, and perform joint training by combining the contrastive learning loss function and the multi-task learning framework to achieve end-to-end cross-modal feature confusion optimization.

[0214] Optionally, the model building module 310 is also used for:

[0215] The speech encoder is responsible for processing speech input and generating a feature representation of the speech;

[0216] The word embedding layer converts text input into an embedded representation;

[0217] The Transformer encoder further processes the speech and text representations, and finally the Transformer decoder generates the target translation output.

[0218] Optionally, token-level feature representations of the speech sequence and text sequence are obtained through a speech encoder and a word embedding layer, respectively; the Transformer encoder encodes the speech encoder output and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module, including:

[0219] By treating speech sequences and text sequences as two independent probability distributions, the optimal transmission problem is formally defined.

[0220] The cost between speech and text features is calculated using Euclidean distance as a preliminary basis for alignment.

[0221] L2 norm normalization is performed on speech and text features respectively to provide standardized input for attention calculation;

[0222] By utilizing the normalized features, attention weight matrices within speech and text sequences are calculated using a self-attention mechanism to capture the dependencies within the sequences.

[0223] By combining the self-attention weights with the original distance matrix, an attention-enhanced distance metric matrix is ​​generated.

[0224] The initial alignment relationship between speech and text tokens is obtained from the enhanced distance matrix by using the minimum distance principle, which provides a foundation for subsequent precise alignment based on the optimal transmission algorithm.

[0225] Optionally, a preliminary alignment relationship between speech and text tokens is obtained from the enhanced distance matrix using the minimum distance principle, providing a foundation for subsequent precise alignment based on the optimal transmission algorithm, including:

[0226] The speech and text token feature vectors that have obtained the initial alignment relationship are respectively constructed as source probability distribution and target probability distribution;

[0227] The distance values ​​in the enhanced distance matrix are used as the transmission cost function between the voice token and the text token.

[0228] The optimal transmission algorithm with relaxed constraints is adopted, which retains only row constraints and removes the strict requirements of column constraints;

[0229] Solve for the optimal transmission matrix under relaxed constraints to obtain the optimal transmission volume allocation from voice token to text token;

[0230] The optimal text token position for each voice token is determined based on the optimal transmission matrix, generating a precise token-level alignment relationship.

[0231] Optionally, the encoded features output by the Transformer encoder are input into the optimal alignment module, attention-enhanced alignment is optimized using a dynamic window strategy, and optimal transfer calculation is performed using the OT computation submodule to generate a cross-modal alignment matrix, including:

[0232] The base window size is calculated based on the length characteristics of the speech and text sequences and adjusted using a scaling factor.

[0233] The window size is dynamically adjusted based on the degree of deviation in the sequence length ratio;

[0234] Based on the position of the source sequence, calculate the window range of the target sequence to ensure that the window can adapt to changes in the length of the speech and text sequences;

[0235] Determine the valid positions based on the padding mask, convert the window mask into binary form, and filter valid alignment candidates.

[0236] The mask is applied to the distance matrix after attention enhancement to maximize the distance to the location outside the window; the optimal target location corresponding to each source location is calculated to obtain the alignment relationship between the speech sequence and the text sequence.

[0237] Optionally, the cross-modal alignment matrix and encoded features are input into the adaptive fusion module, and the speech and text features of the speech sequence and text sequence are fused based on a similarity-based adaptive fusion strategy to obtain a cross-modal representation with natural transitions and semantic coherence, including:

[0238] Calculate the cosine similarity between speech features and text features, and use it as the basis for the mixed weights;

[0239] use The function performs a smooth transformation on the similarity values ​​to obtain the weights used for mixing;

[0240] The mixing coefficient is dynamically adjusted based on similarity to control the fusion ratio of speech and text features;

[0241] The calculated mixing coefficients are used to smoothly fuse speech and text features to obtain cross-modal representations with natural transitions and semantic coherence.

[0242] Optionally, the cross-modal hybrid feature representation is input into the Transformer decoder, and joint training is performed using a contrastive learning loss function and a multi-task learning framework to achieve end-to-end cross-modal feature obfuscation optimization, including:

[0243] High-quality positive and negative sample pairs are constructed by optimizing transmission alignment. Positive sample pairs consist of aligned speech and text features, while negative sample pairs consist of misaligned features.

[0244] The InfoNCE loss function is used to bring positive sample pairs closer together while pushing negative sample pairs further apart.

[0245] By combining contrastive learning loss with multi-task loss functions such as speech translation loss and machine translation loss, a unified optimization objective is formed to achieve end-to-end cross-modal feature confusion optimization.

[0246] In this embodiment of the invention, a cross-modal alignment method based on adaptive optimal transmission is first proposed. Through an innovative attention enhancement mechanism and dynamic window strategy, the alignment accuracy between speech and text is significantly improved, especially for low-resource languages, effectively solving the translation error caused by modal differences.

[0247] Secondly, this invention innovatively introduces an adaptive fusion strategy based on feature similarity, which dynamically adjusts the fusion weights of speech and text features, making cross-modal representation more natural and continuous. In particular, it demonstrates stronger robustness and adaptability when the speech and text lengths differ significantly.

[0248] Finally, the contrastive learning loss function proposed in this invention, combined with a multi-task learning framework, effectively addresses the data scarcity problem in speech translation tasks, particularly enhancing the system's generalization ability in low-resource environments. Through contrastive learning, positive and negative sample pairs are constructed between speech and text features. The InfoNCE loss method narrows the distance between aligned features and widens the distance between unaligned features, thereby enhancing cross-modal representation learning capabilities. Simultaneously, the integration of a multi-task learning framework (including speech translation, machine translation, and cross-modal hybrid tasks) allows the model to fully utilize more data sources for learning, improving its generalization ability and reducing its dependence on large amounts of labeled data.

[0249] Figure 4 This is a schematic diagram of a cross-transport obfuscation device based on adaptive optimal transmission provided in an embodiment of the present invention, as shown below. Figure 4As shown, the straddle-type intermodal mixing device based on adaptive optimal transmission may include the above-mentioned Figure 3 The illustrated straddle-based obfuscation device is an adaptive optimal transmission-based transshipment obfuscation device. Optionally, a straddle-based obfuscation device 410 may include a first processor 2001.

[0250] Optionally, a straddle-based obfuscation device 410 based on adaptive optimal transmission may further include a memory 2002 and a transceiver 2003.

[0251] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0252] The following is combined Figure 4 The components of a straddle-type intermodal obfuscation device 410 based on adaptive optimal transmission are described in detail below:

[0253] The first processor 2001 is a control center of the straddle-type intermodal mixing device 410 based on adaptive optimal transmission. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0254] Optionally, the first processor 2001 can perform various functions of a straddle-and-cross mixing device 410 based on adaptive optimal transmission by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0255] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0256] In a specific implementation, as one example, a straddle-based intermodal obfuscation device 410 based on adaptive optimal transmission may also include multiple processors, for example... Figure 4The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0257] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0258] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via an interface circuit of an adaptive optimal transmission-based cross-transport obfuscation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0259] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0260] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0261] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected via an interface circuit of a cross-transport obfuscation device 410 based on adaptive optimal transmission. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0262] It should be noted that, Figure 4 The structure of the cross-transport obfuscation device 410 based on adaptive optimal transmission shown in the figure does not constitute a limitation on the router. The actual knowledge structure identification device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0263] Furthermore, the technical effects of a straddle-transport obfuscation device 410 based on adaptive optimal transmission can be referred to the technical effects of the straddle-transport obfuscation method based on adaptive optimal transmission described in the above method embodiments, and will not be repeated here.

[0264] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0265] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0266] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0267] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0268] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0269] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A transcontinental obfuscation method based on adaptive optimal transmission, characterized in that, The method includes: S1. Construct the optimal transmission model for a multi-task general framework; wherein, the optimal transmission model includes a speech encoder, a word embedding layer, a Transformer encoder, an optimal alignment module, an adaptive mixing module, and a Transformer decoder; the word embedding layer is set up in parallel with the speech encoder; S2. Token-level feature representations of the speech sequence and text sequence are obtained through the speech encoder and word embedding layer, respectively; the Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module; By treating speech sequences and text sequences as two independent probability distributions, the optimal transmission problem is formally defined. The cost between speech and text features is calculated using Euclidean distance as a preliminary basis for alignment. L2 norm normalization is performed on speech and text features respectively to provide standardized input for attention calculation; By utilizing the normalized features, attention weight matrices within speech and text sequences are calculated using a self-attention mechanism to capture the dependencies within the sequences. By combining the self-attention weights with the original distance matrix, an attention-enhanced distance metric matrix is ​​generated. The initial alignment relationship between speech and text tokens is obtained from the enhanced distance matrix by using the minimum distance principle, which provides a foundation for subsequent precise alignment based on the optimal transmission algorithm; The speech and text token feature vectors that have obtained the initial alignment relationship are respectively constructed as source probability distribution and target probability distribution; The distance values ​​in the enhanced distance matrix are used as the transmission cost function between the voice token and the text token. The optimal transmission algorithm with relaxed constraints is adopted, which retains only row constraints and removes the strict requirements of column constraints; Solve for the optimal transmission matrix under relaxed constraints to obtain the optimal transmission volume allocation from voice token to text token; The optimal alignment of each voice token with the text token is determined based on the optimal transmission matrix, generating a precise token-level alignment relationship; S3. Input the encoded features output by the Transformer encoder into the optimal alignment module, optimize the attention enhancement alignment through the dynamic window strategy, and use the OT calculation submodule to perform optimal transmission calculation to generate a cross-modal alignment matrix. S4. Input the cross-modal alignment matrix and encoded features into the adaptive fusion module. Based on the similarity-based adaptive fusion strategy, fuse the speech and text features of the speech sequence and text sequence to obtain a cross-modal representation with natural transition and semantic coherence. Calculate the cosine similarity between speech features and text features, and use it as the basis for the mixed weights; use The function performs a smooth transformation on the similarity values ​​to obtain the weights used for mixing; The mixing coefficient is dynamically adjusted based on similarity to control the fusion ratio of speech and text features; The calculated mixing coefficients are used to smoothly fuse speech and text features to obtain cross-modal representations with natural transitions and semantic coherence; S5. Input the cross-modal hybrid feature representation into the Transformer decoder, and perform joint training by combining the contrastive learning loss function and the multi-task learning framework to achieve end-to-end cross-modal feature confusion optimization.

2. The method of claim 1, wherein, S1 also includes: The speech encoder is responsible for processing speech input and generating a feature representation of the speech; The word embedding layer converts text input into an embedded representation; The Transformer encoder further processes the speech and text representations, and finally the Transformer decoder generates the target translation output.

3. The method of claim 2, wherein, In step S3, the encoded features output by the Transformer encoder are input into the optimal alignment module. Attention enhancement alignment is optimized using a dynamic window strategy, and optimal transmission calculation is performed using the OT computation submodule to generate a cross-modal alignment matrix, including: The base window size is calculated based on the length characteristics of the speech and text sequences and adjusted using a scaling factor. The window size is dynamically adjusted based on the degree of deviation in the sequence length ratio; Based on the position of the source sequence, calculate the window range of the target sequence to ensure that the window can adapt to changes in the length of the speech and text sequences; Determine the valid positions based on the padding mask, convert the window mask into binary form, and filter valid alignment candidates. The mask is applied to the distance matrix after attention enhancement to maximize the distance to the location outside the window; the optimal target location corresponding to each source location is calculated to obtain the alignment relationship between the speech sequence and the text sequence.

4. The method of claim 3, wherein, In step S5, the cross-modal hybrid feature representation is input into the Transformer decoder, and joint training is performed using a contrastive learning loss function and a multi-task learning framework to achieve end-to-end cross-modal feature obfuscation optimization, including: High-quality positive and negative sample pairs are constructed by using the optimal transmission alignment relationship; positive sample pairs consist of aligned speech and text features, while negative sample pairs consist of misaligned features. The InfoNCE loss function is used to bring positive sample pairs closer together while pushing negative sample pairs further apart. By combining contrastive learning loss with speech translation loss, machine translation loss, and other multi-task loss functions, a unified optimization objective is formed to achieve end-to-end cross-modal feature confusion optimization.

5. An adaptive optimal transport based cross-docking obfuscation apparatus for implementing the adaptive optimal transport based cross-docking obfuscation method of any one of claims 1-4, wherein, The device includes: The model building module is used to construct the optimal transmission model for a multi-task general framework. The optimal transmission model includes a speech encoder, a word embedding layer, a Transformer encoder, an optimal alignment module, an adaptive mixing module, and a Transformer decoder. The word embedding layer is set up in parallel with the speech encoder. The feature encoding module is used to obtain token-level feature representations of speech sequences and text sequences through the speech encoder and word embedding layer, respectively; the Transformer encoder encodes the output of the speech encoder and the text embedding, respectively, to provide semantic feature input for the optimal transmission alignment module; The cross-modal alignment module is used to input the encoded features output by the Transformer encoder into the optimal alignment module, optimize the attention enhancement alignment through a dynamic window strategy, perform optimal transfer calculation using the OT calculation submodule, and generate a cross-modal alignment matrix. The cross-modal representation module is used to input the cross-modal alignment matrix and encoded features into the adaptive fusion module. Based on the similarity-based adaptive fusion strategy, the speech and text features of the speech sequence and text sequence are fused to obtain a cross-modal representation with natural transition and semantic coherence. The optimization module is used to input cross-modal hybrid feature representations into the Transformer decoder, and perform joint training by combining the contrastive learning loss function and the multi-task learning framework to achieve end-to-end cross-modal feature confusion optimization.

6. A transshipment obfuscation device based on adaptive optimal transmission, characterized in that, The straddle-based obfuscation device based on adaptive optimal transmission includes: Processor; memory storing computer-readable instructions that, when executed by the processor, implement any one of the methods in the straddle-transport obfuscation method based on adaptive optimal transmission as described in any one of claims 1-4.

7. A computer readable storage medium characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement any one of the methods in the straddle-transport obfuscation method based on adaptive optimal transmission as described in any one of claims 1-4.

Images