A speech rate self-adaptive audio-visual cross-modal retrieval system and a retrieval method thereof

Abstract

The application provides a speech speed self-adaptive-based audio-visual cross-modal retrieval system and a retrieval method thereof, which comprises the following steps: S1, constructing a data set, randomly changing the speech speed in the current data set to construct a data set containing multiple speech speeds; S2, audio routing, generating a global path through a pre-training model for the input speech, and then dividing the obtained speech and speech speed into multiple small blocks through a dynamic segmentation network to extract features to generate a detailed path; S3, audio fusion, performing speech feature fusion on the two processing paths in step S2 to obtain the final speech feature representation; S4, picture feature, selecting a suitable feature extraction module for the input picture to obtain the final picture feature representation; and S5, feature comparison, calculating a comparison loss according to the final picture feature and the speech feature. The application combines global and segment features to improve the cross-modal retrieval precision, and the application exhibits significant superiority on original and self-built data sets.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology for speech and images, and in particular to a speech-audio cross-modal retrieval system and retrieval method based on speech rate adaptation. Background Technology

[0002] Audio-visual cross-modal retrieval is a core task in human-computer interaction and multimedia understanding, aiming to query related content in one modality (such as audio) and another (such as visual). This technology has broad application prospects in intelligent search engines, e-commerce recommendations, and smart home systems. However, due to the significant semantic gap between audio and visual data, and the inherent time-varying characteristics of speech signals (such as dynamic changes in speech rate), traditional methods struggle to achieve accurate cross-modal alignment.

[0003] Currently, the mainstream methods can be divided into two categories: ① Speech recognition-based methods: These methods convert speech into text and then use pre-trained models (such as CLIP) for cross-modal matching. Their performance is highly dependent on the accuracy of the speech recognition system; when the speech contains dialects or technical terms, the error rate increases exponentially, leading to the loss of semantic information.

[0004] ② End-to-end audio coding-based methods: These methods employ a two-stream framework to directly align spectrograms and image features. While these methods have made some progress, they suffer from the following limitations: they ignore the dynamics of speech rate, leading to significant fluctuations in information content per unit time. Traditional models use fixed-length windows to extract features, which has significant drawbacks when processing speech at different rates. Specifically, for high-speed speech, a fixed window may not cover complete semantic units, resulting in information loss; while for low-speed speech, the semantic content within the fixed window may be too sparse, leading to information redundancy and increased noise interference.

[0005] Changes in speech rate lead to two key issues: ① Semantic coherence is disrupted: When the same content is spoken at different speeds, the semantic boundaries of audio segments become blurred, and fixed segmentation strategies cannot adapt to changes in information density. ② Decreased feature discrimination power: Intra-class differences (feature shift of the same semantics at different speech rates) increase, while inter-class differences (feature confusion of different semantics at similar speech rates) decrease, exacerbating the difficulty of model discrimination. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a speech rate adaptive audio-visual cross-modal retrieval system and its retrieval method, which solves the problem that the difficulty in audio representation due to different speaking rates in the prior art limits the retrieval performance.

[0007] To achieve the above and other related objectives, this invention provides a cross-modal audio-visual retrieval system based on speech rate adaptation, comprising: a speech feature extraction module for extracting speech features; a speech rate processing module for determining segmentation strategies based on different speech rates of the current speech, wherein the speech rate processing module obtains speech rate labels for the current audio through a lightweight convolutional neural network based on the Mel-Frequency Cepstral coefficients of the audio, corresponding to three different speech rate scales, and determines the number of audio segments based on the speech rate labels of the current audio; a speech feature fusion module for fusing the speech features of the detail path and the global path to obtain the final speech features; an image feature processing module for obtaining the feature representation of the image; and a processing module for calculating a comparison loss between the language features obtained by the speech feature fusion module and the image features obtained by the image feature processing module.

[0008] A detection method for a speech rate-adaptive audiovisual cross-modal retrieval system as described above includes the following steps: S1. Construct a dataset by randomly varying the speech rate in the current dataset to create a dataset containing multiple speech rates. S2, Audio Segmentation: The input speech is processed by a pre-trained model to generate a global path, and then a dynamic segmentation network is used to segment the obtained speech and speech rate into multiple small blocks and extract features to generate detailed paths. S3. Audio fusion: The two processing paths in step S2 are fused to obtain the final speech feature representation. S4. Image features: Select an appropriate feature extraction module for the input image to obtain the final image feature representation; S5. Feature comparison: Calculate the contrast loss based on the final image and speech features.

[0009] By adopting the above technical solution, the length of audio segments can be dynamically adjusted according to the speaking rate of the audio, capturing phoneme-level information of different segments and fusing it with the overall semantic clues to form an audio representation to perform the retrieval process. This effectively solves the problem that traditional fixed-window audio feature descriptors are difficult to adapt to dynamic changes in speaking rate, and reduces information loss caused by fast speaking speed and information redundancy caused by slow speaking speed.

[0010] In one embodiment of the present invention, in step S1, the audio speech rate is adjusted from slow to fast using the Flickr8k audio title corpus and SpokenCOCO as the original data basis, and the speed-adjusted audio is rematched with the original corresponding visual image to form a multi-speed audiovisual pair and construct an extended dataset.

[0011] In one embodiment of the present invention, in step S2, a pre-trained HuBERT is used to extract global features to generate a global pathway, the mathematical formula of which is defined as: A i and Representing i respectively th Audio samples and corresponding uniform speech rate feature vectors.

[0012] In one embodiment of the present invention, in step S2, the speech rate of the current speech is obtained through a speech rate classification network in the detail path. The extracted speech rate and the speech are entered into a dynamic segmentation network and divided into n small blocks. Each small block is individually processed by HuBERT to extract features, and then the blocks are concatenated together and processed by Transformer to obtain the features of the detail path.

[0013] In one embodiment of the present invention, in step S2, the adaptive speech rate audio features in the detail path... The mathematical formula is defined as follows: ,in express The number of segments in the audio sample.

[0014] In one embodiment of the present invention, in step S1, the audio speech rate is adjusted to three magnifications from slow to fast, namely 0.5x, 1x, and 2x; The defining formula is: The labels for r are obtained by a lightweight convolutional neural network based on audio Mel-FrequencyCepstral coefficients, corresponding to three multiples of the audio speech rate.

[0015] In one embodiment of the present invention, in step S2, the input feature of the speech rate detection network structure in the detail path is a Mel spectrogram normalized to 256x256 through bilinear interpolation, and the output feature is a 3-dimensional logarithmic vector corresponding to three speech rate categories: 0.5, 1, and 2. The final prediction obtains the category index through the index of the maximum value. The mathematical expression of the detail path is: ,in Let represent the t-th segment of the i-th audio sample.

[0016] In one embodiment of the present invention, in step S3, a dynamic weighted fusion model is used to determine the magnitude of each weight in the tensors of the detail path and the global path to form the final representation of the audio. The final expression a is: .

[0017] In one embodiment of the present invention, in step S4, the CLIP model is used to extract the feature representation of the image during image feature extraction, and its mathematical formula is defined as follows: Where Vi and vi represent the visual sample and corresponding feature vector of ith, respectively, CLIP V iT-L / 14 (·) De-annotate the visual encoder based on the pre-trained CLIP.

[0018] As described above, the speech rate-adaptive cross-modal audiovisual retrieval system and method of the present invention have the following beneficial effects: By constructing a multi-speech-rate audiovisual dataset based on Flickr8k and SpokenCOCO, a dual-stream collaborative network architecture is proposed: speech is dynamically segmented through a speech rate detection network, global and segment features are extracted by combining HuBERT, and after dynamic weighted fusion, cross-modal retrieval is performed with the visual features of CLIP, and bidirectional InfoNCE loss is used to optimize alignment accuracy. Experiments show that the average recall rate of this method is better than that of traditional models on both fixed-speech-rate and multi-speech-rate datasets, verifying the effectiveness of the adaptive speech rate alignment strategy in improving the accuracy and adaptability of cross-modal retrieval. Attached Figure Description

[0019] Figure 1 The diagram shown is a schematic representation of the overall process disclosed in an embodiment of the present invention.

[0020] Figure 2 The diagram shows the network structure of the dynamic weight fusion module disclosed in Embodiment 2 of the present invention. Detailed Implementation

[0021] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0022] Please see Figures 1 to 2 It should be understood that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes to aid those skilled in the art and to facilitate understanding and reading. They are not intended to limit the scope of the invention and therefore have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effectiveness and purpose of the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity and not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention's implementation.

[0023] Example 1: like Figure 1 As shown, this embodiment provides a cross-modal audiovisual retrieval system based on speech rate adaptation, including: The speech feature extraction module is used to extract speech features; The speech rate processing module determines the segmentation strategy based on the different speech rates of the current speech. The speech feature fusion module fuses the speech features of the detail path and the global path to obtain the final speech features; The image feature processing module is used to obtain the feature representation of the image. In this embodiment, the image feature processing module includes an image feature extractor (CLIP). The processing module calculates the comparison loss by combining the language features obtained from the speech feature fusion module with the image features from the image feature processing module.

[0024] The speech feature extraction module is a speech feature extractor (HuBERT), which extracts speech feature data from selected samples.

[0025] The speech rate processing module includes a speech rate classification network and a dynamic segmentation network. The speech feature fusion module is a dynamic weight fusion network. The speech rate processing module adjusts the speech rate of the selected speech samples, adjusting the audio speech rate to three levels: 0.5 (slow), 1 (original), and 2 (fast), to simulate the drastic changes in speech rate in real-world scenarios.

[0026] During the adjustment process, the semantic content of the audio remains unchanged, and only the compression or stretching of the time dimension is changed to avoid introducing additional noise or semantic distortion.

[0027] The altered audio is rematched with the original corresponding visual image to form multi-speed audio-visual pairs, thus constructing an extended dataset.

[0028] The input speech is divided into two paths: one is a global path, which directly uses the pre-trained HuBERT to extract global features. The other speech stream is processed by a speech rate classification network to obtain the speech rate of the current speech. The extracted speech rate and the speech are then entered into a dynamic segmentation network to be segmented into n small blocks. Each small block is processed separately to extract features using HuBERT, and then the blocks are concatenated together and processed by a Transformer to obtain the features of the details pathway. The two speech features are then processed by a dynamic weight fusion network to obtain the final speech features. For the input image features, CLIP is used to extract the image features. Finally, a comparison loss is applied to the final set of speech features and image features.

[0029] Example 2: A retrieval method for a cross-modal audiovisual retrieval system based on speech rate adaptation as described in Embodiment 1 includes the following steps: S1. Construct a dataset by randomly varying the speech rate in the current dataset to create a dataset containing multiple speech rates. S2, Audio Segmentation: The input speech is processed by a pre-trained model to generate a global path, and then a dynamic segmentation network is used to segment the obtained speech and speech rate into multiple small blocks and extract features to generate detailed paths. S3. Audio fusion: The two processing paths in step S2 are fused to obtain the final speech feature representation. S4. Image features: Select an appropriate feature extraction module for the input image to obtain the final image feature representation; S5. Feature comparison: Calculate the contrast loss based on the final image and speech features. In step S1, the Flickr8k audio title corpus and SpokenCOCO are used as the original data base to adjust the audio speed from slow to fast at multiple magnifications, and the speed-changed audio is rematched with the original corresponding visual image to form a multi-speed audio-visual pair and construct an extended dataset. In this embodiment, the audio speed is adjusted to three magnifications from slow to fast: 0.5x (slow), 1x (original), and 2x (fast).

[0030] In step S2, pre-trained Hubert is used to extract global features to generate global pathways. The global pathways correspond to the feature representation branches of the unified speech rate, and their mathematical formula is defined as Equation (1): (1) Where A i and Representing i respectively th Audio samples and corresponding uniform speech rate feature vectors.

[0031] In step S2, the speech rate of the current speech is obtained through the speech rate classification network in the detail path. The speech rate and the speech are extracted and entered into the dynamic segmentation network to be segmented into n small blocks. Each small block is used to extract features separately through HuBERT and then concatenated together to obtain the features of the detail path through Transformer. In the detail pathway, the speech rate detection network structure takes Mel spectrograms as input features, which are normalized to 256x256 by bilinear interpolation. The output features are 3-dimensional logarithmic vectors, corresponding to three speech rate categories: 0.5, 1, and 2. The final prediction obtains the category index by the index of the maximum value.

[0032] The detail path corresponds to the feature representation branch of the adaptive speech rate, and the mathematical formula for the adaptive speech rate audio feature is defined as equation (2): (2) in express The number of audio sample segments, The mathematical formula is defined as shown in equation (3): (3) The labels for r are obtained by a lightweight convolutional neural network based on audio Mel-Frequency Cepstral coefficients, corresponding to three different scaling factors of the audio speech rate.

[0033] In equation (2), (4) in Let represent the t-th segment of the i-th audio sample; then, through step S3, the final audio representation is formed using the Dynamic Weighted Fusion (DWF) model, and its final expression 'a' is: (5) Its architecture is as follows Figure 2 As shown.

[0034] In step S4, visual modal features are extracted using the CLIP-pretrained ViT-L / 14, and its mathematical formula is defined as shown in equation (6): (6) Among them Vi and v i Representing i respectively th Visual samples and corresponding feature vectors, CLIP V iT-L / 14 (·) De-annotate the visual encoder based on the pre-trained CLIP.

[0035] Then, in step S5, the speech features from step S3 and the image features from step S4 are compared and lost together to perform cross-modal retrieval, and the alignment accuracy is optimized using bidirectional InfoNCE loss.

[0036] In summary, this invention addresses the semantic representation challenge caused by speech rate variations in cross-modal audiovisual retrieval. It constructs a multi-rate audiovisual dataset based on Flickr8k and SpokenCOCO, and proposes a dual-stream collaborative network architecture: a speech rate detection network dynamically segments speech, and HuBERT is used to extract global and segmental features. These features are then dynamically weighted and fused with CLIP visual features for cross-modal retrieval. A bidirectional InfoNCE loss is employed to optimize alignment accuracy. Experiments show that this method outperforms traditional models in average recall on both fixed-rate and multi-rate datasets, validating the effectiveness of the adaptive speech rate alignment strategy in improving the accuracy and adaptability of cross-modal retrieval.

[0037] Therefore, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0038] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A cross-modal audiovisual retrieval system based on speech rate adaptation, characterized in that, include: The speech feature extraction module is used to extract speech features; The speech rate processing module adjusts the speech rate multiplier and determines the segment segmentation strategy to generate detailed and global paths based on the different speech rates of the current speech. The speech rate processing module obtains the speech rate label of the current audio through a lightweight convolutional neural network based on the Mel-Frequency Cepstral coefficient of the audio, which corresponds to the three multipliers of the audio speech rate, and determines the number of audio segments based on the speech rate label of the current audio. The speech feature fusion module fuses the speech features of the detail path and the global path to obtain the final speech features; Image feature processing module, used to obtain the feature representation of an image; The processing module calculates the comparison loss by combining the language features obtained from the speech feature fusion module with the image features from the image feature processing module.

2. A retrieval method for a cross-modal audiovisual retrieval system based on speech rate adaptation as described in claim 1, characterized in that: Includes the following steps: S1. Construct a dataset by randomly varying the speech rate in the current dataset to create a dataset containing multiple speech rates. S2, Audio Segmentation: The input speech is processed by a pre-trained model to generate a global path, and then a dynamic segmentation network is used to segment the obtained speech and speech rate into multiple small blocks and extract features to generate detailed paths. S3. Audio fusion: The two processing paths in step S2 are fused to obtain the final speech feature representation. S4. Image features: Select an appropriate feature extraction module for the input image to obtain the final image feature representation; S5. Feature comparison: Calculate the contrast loss based on the final image and speech features.

3. The retrieval method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 2, characterized in that: In step S1, the Flickr8k audio title corpus and SpokenCOCO are used as the original data base to adjust the audio speed from slow to fast at various magnifications, and the speed-changed audio is rematched with the original corresponding visual image to form a multi-speed audiovisual pair and construct an extended dataset.

4. The retrieval method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 2, characterized in that: In step S2, pre-trained HuBERT is used to extract global features and generate global pathways, the mathematical formula of which is defined as: A i and Representing i respectively th Audio samples and corresponding uniform speech rate feature vectors.

5. The detection method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 3, characterized in that: In step S2, the speech rate of the current speech is obtained through a speech rate classification network in the detail path. The extracted speech rate and speech are entered into a dynamic segmentation network and divided into n small blocks. Each small block is individually processed by HuBERT to extract features, and then the blocks are concatenated together and processed by Transformer to obtain the features of the detail path.

6. The detection method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 5, characterized in that: In step S2, the adaptive speech rate audio features in the detail path The mathematical formula is defined as: ,in express The number of audio sample segments; the mathematical formula for adaptive speech rate audio features. ,in Let represent the t-th segment of the i-th audio sample.

7. The detection method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 6, characterized in that: In step S1, the audio speaking speed is adjusted to three magnifications from slow to fast: 0.5x, 1x, and 2x. The defining formula is: The labels for r are obtained by a lightweight convolutional neural network based on audio Mel-Frequency Cepstral coefficients, corresponding to three different scaling factors of the audio speech rate.

8. The detection method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 7, characterized in that: In step S2, the input feature of the speech rate detection network structure in the detail path is the Mel spectrogram, which is normalized to 256x256 by bilinear interpolation. The output feature is a 3-dimensional logarithmic vector, corresponding to three speech rate categories: 0.5, 1, and 2. Finally, the category index is obtained by the index of the maximum value.

9. The detection method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 8, characterized in that: In step S3, a dynamic weighted fusion model is used to determine the magnitude of each weight in the tensors of the detail path and the global path to form the final audio representation. The final expression a is: .

10. The detection method of the audiovisual cross-modal retrieval system based on speech rate adaptation according to claim 2, characterized in that: In step S4, the CLIP model is used to extract the feature representation of the image in image feature extraction. Its mathematical formula is as follows: Where Vi and vi represent the visual sample and corresponding feature vector of ith, respectively, CLIP V iT-L / 14 (·) De-annotate the visual encoder based on the pre-trained CLIP.

Images