A deep semantic understanding method based on cross-modal model

Abstract

The present invention discloses a method for deep semantic understanding based on a cross-modal model, comprising calling a WIT dataset to pre-train a text encoder and an image encoder; calling a dataset to pre-train an audio feature extraction sub-network; calling a dataset to perform transfer training on the pre-trained text encoder, image encoder, and audio feature extraction sub-network to obtain a cross-modal model after weight transfer learning; calling a dataset and a dataset to optimize the audio feature extraction sub-network in the cross-modal model after weight transfer learning to obtain a cross-modal model after weight optimization; running the cross-modal model after weight optimization, outputting the association information between text data, image data, and audio data in a multi-source dataset, and completing deep semantic understanding of the multi-source dataset. The present invention solves the problem that current multi-source data intelligent detection and recognition algorithms do not effectively utilize multiple modal data, and the analysis results are poorly secure and stable.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal semantic understanding, and specifically relates to a deep semantic understanding method based on a cross-modal model. Background Art

[0002] With the continuous development of high-tech technologies such as the internet and multimedia, multi-source data integrating images, text, and audio has gradually become the mainstream information dissemination medium, closely related to people's real-life situations. Currently, research in single-modality fields such as computer vision, natural language processing, and speech recognition has made significant progress. However, how to further develop cross-modal semantic understanding and reasoning between vision, text, and audio, and thus narrow the semantic gap between modalities, has become a hot topic.

[0003] Deep learning is an artificial intelligence algorithm that relies on sample labeled data. It learns data representations and continuously trains the algorithm to improve its data detection and recognition capabilities. Deep learning primarily addresses intelligent detection and recognition problems, enabling timely and accurate detection and recognition based on current data information, making it ideally suited for intelligent data analysis scenarios. However, current challenges with deep learning algorithms include insufficient ability to jointly learn from different types of data, an inability to efficiently fuse modal information, an inability to accurately correlate and align modalities, and difficulty effectively mining and understanding semantic information within each modality. These issues raise concerns about the security of deep learning detection and recognition results, making them difficult to accurately guide applications.

[0004] Cross-modal algorithms are recognized as machine learning algorithms with highly secure detection and recognition results. Through joint representation learning, they map multi-source data into a unified cross-modal vector space, combining information from multiple modalities for target detection or regression. Their advantages are security and stability, and they can be used to solve detection and recognition problems for a variety of data types, closely aligned with human cognition. However, common cross-modal algorithms have the disadvantage of failing to effectively integrate the three primary modalities used by humans to understand and describe the real world. They are prone to overfitting, suffer from poor security and stability under noise perturbations, and exhibit poor learning outcomes. Summary of the Invention

[0005] One of the purposes of the present invention is to provide a deep semantic understanding method based on a cross-modal model, which solves the problem that the current multi-source data intelligent detection and recognition algorithm does not effectively utilize multiple modal data and the analysis results are poor in security and stability.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A deep semantic understanding method based on a cross-modal model, wherein the cross-modal model includes a text encoder, an image encoder, and an audio feature extraction subnetwork. The deep semantic understanding method based on the cross-modal model includes:

[0008] Call the WIT dataset to pre-train the text encoder and image encoder;

[0009] Use the ImageNet dataset to pre-train the audio feature extraction sub-network;

[0010] The AudioSet dataset is used to perform transfer training on the pre-trained text encoder, image encoder, and audio feature extraction sub-networks to obtain a cross-modal model after weight transfer learning.

[0011] The ESC-50 and AudioSet datasets are used to optimize the audio feature extraction subnetwork in the cross-modal model after weight transfer learning to obtain a cross-modal model with optimized weights.

[0012] Run the cross-modal model after weight optimization to output the correlation information between text data, image data and audio data in the multi-source dataset, and complete the deep semantic understanding of the multi-source dataset.

[0013] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0014] Preferably, calling the WIT dataset to pre-train the text encoder and the image encoder includes:

[0015] The text encoder extracts text feature vectors of text data in the WIT data set;

[0016] The image encoder extracts image feature vectors of image data in the WIT data set;

[0017] Calculate the cosine similarity between the text data and the image data in the text-image pair based on the normalized text feature vector and the image feature vector;

[0018] The loss value of the text data and the loss value of the image data are calculated based on the cosine similarity, and the text encoder and the image encoder are updated.

[0019] Preferably, the calculating the loss value of the text data and the loss value of the image data based on the cosine similarity, and updating the text encoder and the image encoder, comprises:

[0020] The loss value of the text data is calculated as follows:

[0021]

[0022] Where, loss t is the loss value of text data, y t is the true label of the text data based on the text-image pair, is the predicted probability value of the text data based on the text-image pair, that is, the cosine similarity between the text data and the image data in the text-image pair;

[0023] The loss value of the image data is calculated as follows:

[0024]

[0025] Where, loss i is the loss value of the image data, y i is the true label of the image data based on the text-image pair, is the predicted probability value of the image data based on the text-image pair, that is, the cosine similarity between the text data and the image data in the text-image pair;

[0026] The loss value of the text-image pair is calculated as follows:

[0027]

[0028] Where, loss ti is the loss value of the text-image pair;

[0029] Backpropagation is performed based on the loss value of the text-image pair to update the weights of the text encoder and image encoder.

[0030] Preferably, calling the ImageNet dataset to pre-train the audio feature extraction sub-network includes:

[0031] Use RandomErasing and Mixup methods to perform data enhancement on the ImageNet dataset;

[0032] The audio feature extraction subnetwork is pre-trained using the data-augmented ImageNet dataset.

[0033] Preferably, calling the AudioSet dataset to perform migration training on the pre-trained text encoder, image encoder, and audio feature extraction subnetwork includes:

[0034] Convert the audio data in the AudioSet dataset into a spectrogram through short-time Fourier transform;

[0035] Mapping the spectrogram onto three input channels according to frequency bands to obtain a three-channel image;

[0036] Take an audio-text pair from the AudioSet dataset, use the three-channel image of the audio data in the audio-text pair as input, and use the text data in the audio text as labels to perform extended training on the pre-trained audio feature extraction subnetwork;

[0037] The expanded trained audio feature extraction subnetwork is combined with the pre-trained text encoder and image encoder as the cross-modal model to be trained. The audio-text-image pairs in the AudioSet dataset are taken to perform migration training on the cross-modal model.

[0038] Preferably, the method of obtaining audio-text-image pairs from the AudioSet dataset and performing migration training on the cross-modal model using the audio-text-image pairs includes:

[0039] The text encoder extracts a text feature vector of text data in the audio-text-image pair;

[0040] The image encoder extracts an image feature vector of image data in an audio-text-image pair;

[0041] The audio feature extraction subnetwork extracts audio feature vectors from the three-channel image corresponding to the audio data in the audio-text image pair;

[0042] Calculate the cosine similarity between the text data and the image data, the cosine similarity between the audio data and the text data, and the cosine similarity between the image data and the audio data based on the normalized text feature vector, the image feature vector, and the audio feature vector;

[0043] The loss value is calculated based on the cosine similarity between text data and image data, the cosine similarity between audio data and text data, and the cosine similarity between image data and audio data, and the cross-modal model is updated.

[0044] Preferably, the calculating of the loss value and updating the cross-modal model based on the cosine similarity between text data and image data, the cosine similarity between audio data and text data, and the cosine similarity between image data and audio data comprises:

[0045] Calculate the text image loss value based on the cosine similarity of text data and image data;

[0046] Calculate the audio-text loss value based on the cosine similarity between audio data and text data;

[0047] Calculate the image audio loss value based on the cosine similarity of the image data and the audio data;

[0048] The text image loss value, audio text loss value and image audio loss value are added and averaged as the final loss value to update the cross-modal model.

[0049] Preferably, calling the ESC-50 dataset and the AudioSet dataset to optimize the training of the audio feature extraction subnetwork in the cross-modal model after weight transfer learning includes:

[0050] Take the audio-text pairs from the ESC-50 dataset, freeze the weights of the text encoder and image encoder in the cross-modal model, train the cross-modal model with the audio-text pairs, and adjust the weights of the audio feature extraction subnetwork;

[0051] Take the audio-text pairs in the AudioSet dataset, freeze the weights of the image encoder in the cross-modal model, train the cross-modal model with the audio-text pairs, and adjust the weights of the audio feature extraction subnetwork and the text encoder.

[0052] The present invention provides a deep semantic understanding method based on a cross-modal model. According to the cross-modal model, different modal data are jointly trained, which can align the image, audio and text feature domains, and use the cosine similarity function to compare the similarity of the features of different modal data, thereby improving the generalization, representation and reasoning capabilities of the model, thereby greatly improving the security of the detection results of the cross-modal deep learning algorithm, and solving the problems that single-modal deep learning algorithms cannot achieve cross-modal pre-training, cross-modal retrieval and weak security.

[0053] The second purpose of the present invention is to provide a deep semantic understanding device based on a cross-modal model, which solves the problem that the current multi-source data intelligent detection and recognition algorithm does not effectively utilize multiple modal data and the analysis results are poor in security and stability.

[0054] To achieve the above object, the technical solution adopted by the present invention is:

[0055] A deep semantic understanding device based on a cross-modal model includes a processor and a memory storing a plurality of computer instructions. When the computer instructions are executed by the processor, the steps of the deep semantic understanding method based on the cross-modal model are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is an overall flow chart of the deep semantic understanding method based on the cross-modal model of the present invention;

[0057] Figure 2 This is a specific flow chart of the deep semantic understanding method based on the cross-modal model of the present invention;

[0058] Figure 3 Schematic diagram of an implementation of the cosine similarity function of the present invention;

[0059] Figure 4 A schematic diagram of an implementation structure of the cross-modal model established for the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0062] like Figure 1 As shown, the overall process of the deep semantic understanding method based on the cross-modal model in this embodiment is: use multiple data sets to pre-train the cross-modal model (cross-modal deep learning model), complete the cross-modal model parameter initialization, use a data set containing three modalities to migrate and optimize the pre-trained model parameters, and perform intelligence analysis testing and application on the optimized model.

[0063] Currently, mainstream data types mainly include text data, image data, and audio data. Therefore, this embodiment establishes a cross-modal model including a text encoder, an image encoder, and an audio feature extraction sub-network. In other embodiments, if it is necessary to understand and parse other types of data, the corresponding recognition network can be added to the cross-modal model based on the logic of this application.

[0064] Text encoders and image encoders in deep learning are methods used to convert text and image information into vector representations that machines can understand. Commonly used text encoders include BERT and GPT, while commonly used image encoders include VGG, ResNet, and Inception. The use of text encoders and image encoders can more efficiently process large amounts of text and image data, improving the accuracy and efficiency of deep learning.

[0065] Specifically, such as Figure 2 As shown, the cross-modal model-based deep semantic understanding method of this embodiment includes the following steps:

[0066] Step S1: Call the WIT dataset to pre-train the text encoder and image encoder.

[0067] The WIT (WebImageText) dataset contains both images and text, and can be used to train image retrieval and text recognition models. It helps machine learning models better understand the relationship between images and text. This example uses text-image pairs from the WIT dataset to pre-train the text encoder and image encoder of a deep learning model using two modalities.

[0068] The WIT dataset contains N text labels. During the training process, each text label is encoded by the model's text encoder to obtain a corresponding text feature vector containing multiple label categories stored in a one-dimensional vector data format; each image is encoded by the image encoder to obtain a corresponding image feature vector stored in a one-dimensional vector data format, where the text feature vector is used as a row and the image feature vector is used as a column to form a two-dimensional feature vector. The cosine similarity is calculated, as shown in the following example: Figure 3 The one with the highest similarity is the Top-1 predicted category.

[0069] Step S1.1: The text encoder extracts text feature vectors of text data in the WIT dataset.

[0070] The text feature vector output by the text encoder is as follows:

[0071] T=[T1,T2,…T i ,…T M ]

[0072] Where T is the text feature vector in one-dimensional vector data format, M is the total number of text features in the text feature vector, and T i is the i-th text feature.

[0073] Step S1.2: The image encoder extracts the image feature vector of the image data in the WIT dataset.

[0074] The image feature vector output by the image encoder is as follows:

[0075] I=[I1,I2,…I j ,…I N ]

[0076] Where I is the image feature vector in one-dimensional vector data format, N is the total number of image features in the image feature vector, and I j is the jth image feature.

[0077] Step S1.3: Calculate the cosine similarity between the text data and the image data in the text-image pair based on the normalized text feature vector and image feature vector.

[0078] Map two different unimodal features to the same vector space and calculate L2 normalization. The L2 normalization expression is as follows:

[0079] x=[x1,x2,…,x L ]

[0080] y=[y1,y2,…,y L ]

[0081]

[0082] In the formula, x represents a one-dimensional feature vector of length L, x i is the i-th value in the one-dimensional feature vector x, and y represents the one-dimensional feature vector after L2 normalization. Substitute the text feature vector and image feature vector into the normalization formula respectively to obtain the normalized text feature vector T ′ =[T1 ′ ,T2 ′ ,…T i ′ ,…T ′ M ] and image feature vector I ′ =[I1 ′ ,I2 ′ ,…I j ′ ,…I ′ N ].

[0083] The cosine similarity function is used to calculate the probability that the original non-normalized predictions of the two modalities are similar. The cosine similarity function expression is as follows:

[0084]

[0085] In the formula, cosine similarities(T ′ ,I ′ ) is the cosine similarity of the normalized text feature vector and the image feature vector, indicating the probability of whether the two are similar. The value range of the cosine similarity function is [-1, 1]. However, in this application, the value of cosine similarity is between 0 and 1, where 0 is completely dissimilar and 1 is completely similar. The closer the value is to 1, the higher the similarity. In addition, in a text-image pair, the cosine similarity of the text data and the image data is the same as the cosine similarity of the image data and the text data, i.e., cosine similarities (I ′ ,T ′ )=cosine similarities(T ′ ,I ′ ).

[0086] Step S1.4: Calculate the loss value of the text data and the loss value of the image data based on the cosine similarity, and update the text encoder and the image encoder.

[0087] cosine similarities(T ′ ,I ′ ) and GroundTruth (GroundTruth represents the true target label, represented by a unit matrix, the highest similarity between paired samples is 1, and the others are 0) to calculate the cross entropy loss function. The cross entropy loss function expression is as follows:

[0088] Step S1.4.1. Calculate the loss value of the text data as follows:

[0089]

[0090] Where, loss t is the loss value of the text data, that is, the difference between the real sample label of the image and the predicted probability, y t is the true label of the text data based on the text-image pair, is the predicted probability value of the text data based on the text-image pair, that is, the cosine similarity calculated in step S1.3, with a value of [0, 1].

[0091] Step S1.4.2: Calculate the loss value of the image data as follows:

[0092]

[0093] Where, loss i is the loss value of the image data, that is, the difference between the real sample label and the predicted probability of the image, y i is the true label of the image data based on the text-image pair, is the predicted probability value of the image data based on the text-image pair, that is, the cosine similarity calculated in step S1.3, and its value is [0,1].

[0094] Step S1.4.3: Calculate the loss value of the text-image pair as follows:

[0095]

[0096] Where, loss ti is the loss value of the text-image pair.

[0097] Step S1.4.4: Back propagation is performed based on the loss value of the text-image pair to update the weights of the text encoder and the image encoder. It should be noted that back propagation of the deep learning model is a conventional model update method and will not be described in detail in this embodiment.

[0098] Step S2: Call the ImageNet dataset to pre-train the audio feature extraction sub-network.

[0099] Initializing model weights involves using supervised learning to obtain a task-independent pre-trained model from large-scale data. This is an application of transfer learning, which can transfer knowledge learned in open domains to downstream tasks to improve low-resource tasks. The audio feature extraction subnetwork is primarily implemented using image-based data augmentation algorithms, a ResNet network, and an attention mechanism algorithm.

[0100] Step S2.1: Use the RandomErasing method and the Mixup method to perform data enhancement on the ImageNet dataset.

[0101] Data augmentation (such as RandomErasing, Mixup, and CutMix) is a lightweight method for improving the generalization ability of network models. It does not require any additional parameters or memory consumption and can be integrated with various neural network models without changing the learning strategy. In this example, RandomErasing and Mixup are selected as the data augmentation algorithms for the pre-trained audio feature extraction subnetwork.

[0102] This example performs two data augmentations, RandomErasing and Mixup, on the ImageNet dataset. The expression of the RandomErasing method is as follows:

[0103] S=W×H

[0104]

[0105] r e ∈(r1,r2)

[0106]

[0107]

[0108] P=(x e ,y e )

[0109] I e =(x e ,y e ,x 1e +W e ,y e +H e )

[0110] Where, I e To randomly erase a rectangular area in a random image, S is the image area, W is the image width, H is the image height, Se To randomly initialize the size of the “erasing rectangle”, (S l ,S h ) indicates S e Length and height, r e ∈(r1,r2) represents S e The aspect ratio, and Represents rectangle I e The height and width of the image, P represents a random point in the image, (x e ,y e ) represents the horizontal and vertical coordinates of the point, I e =(x e ,y e ,x e +W e ,y e +H e ) indicates that the rectangular area is expanded from this point. Within the selected area, each pixel is assigned a random value in [0,255].

[0111] The expression of the Mixup method is as follows:

[0112]

[0113]

[0114] In the formula, (x i ,y i ) and (x j ,y j ) are two randomly selected samples and corresponding labels from the same batch of input data, λ is the number randomly sampled from the beta distribution, λ∈[0,1].

[0115] Step S2.2: Pre-train the audio feature extraction subnetwork using the data-enhanced ImageNet dataset, and save the weights of the audio feature extraction subnetwork trained using the ImageNet dataset.

[0116] The ResNet network in this embodiment integrates the attention mechanism, which is expressed as follows:

[0117]

[0118] Where x is the signal processed by the convolution filter, is the output of the attention module and the input of the next layer of network, L i (x) is the input of the i-th layer network, A i (x) is the input of the i-th attention module, L i (x) and A i(x) These two inputs are identical.

[0119] When pre-training the audio feature extraction sub-network, conventional optimization methods can be used, such as the gradient descent algorithm, and the optimizer can be directly used for optimization.

[0120] Step S3: Call the AudioSet dataset to perform migration training on the pre-trained text encoder, image encoder, and audio feature extraction subnetwork to obtain a cross-modal model after weight transfer learning.

[0121] First, we select audio data from the AudioSet dataset to perform extended training on the pre-trained audio feature extraction sub-network to improve the audio detection performance of the cross-modal algorithm model. Then, we integrate the extended trained audio feature extraction sub-network into the cross-modal deep learning algorithm model, and use the audio clips, corresponding video frames, and specified text labels from the AudioSet dataset to complete the migration training of the entire cross-modal deep learning algorithm model in conjunction with the image encoder and text encoder. The structure of the cross-modal deep learning algorithm model is as follows: Figure 4 The specific training steps are as follows:

[0122] Step S3.1: Convert the audio data in the AudioSet dataset into a spectrogram through short-time Fourier transform. The expression is as follows:

[0123]

[0124] Where X(τ,ω) is the amplitude and phase of the fundamental sinusoidal frequency ω at different time points τ in the time domain signal x, τ is the different time points in the time domain signal, ω is the fundamental sinusoidal frequency, x[n] represents the corresponding basis coordinates, and ω[n-τ] represents the analysis window function.

[0125] In order to reduce the spectrum disturbance caused by framing, the Blackman-Harris window function is used to reduce the amount of noise in the spectrum. The window function expression is as follows:

[0126]

[0127] Wherein, a0=0.35875, a1=0.48829, a2=0.14128, a3=0.01168, t represents time, N is the total window length, W[t] is the weighted value of the time function, and k=0, 1, 2, ..., N-1.

[0128] Step S3.2: Map the spectrum image onto three input channels according to the frequency band to obtain a three-channel image.

[0129] This embodiment maps the spectrogram onto three input channels along its frequency axis to obtain a three-channel image, which is divided into three frequency bands: low (0.00-7.35 kHz), medium (7.35-14.70 kHz), and high (14.7-22.05 kHz).

[0130] Step S3.3: Take an audio-text pair from the AudioSet dataset, use the three-channel image of the audio data in the audio-text pair as input, and use the text data in the audio text as labels to perform extended training on the pre-trained audio feature extraction subnetwork. When performing extended training on the audio feature extraction subnetwork, conventional optimization methods, such as gradient descent, can be used.

[0131] Step S3.4: Expand the trained audio feature extraction subnetwork and combine it with the pre-trained text encoder and image encoder as the cross-modal model to be trained. Take the audio-text-image pairs in the AudioSet dataset and perform migration training on the cross-modal model using the audio-text-image pairs.

[0132] Step S3.4.1: The text encoder extracts the text feature vector of the text data in the audio-text-image pair.

[0133] The text feature vector output by the text encoder is as follows:

[0134] T=[T1,T2,...T i ,...T M ]

[0135] Where T is the text feature vector in one-dimensional vector data format, M is the total number of text features in the text feature vector, and T i is the i-th text feature.

[0136] Step S3.4.2: The image encoder extracts the image feature vector of the image data in the audio-text-image pair.

[0137] The image feature vector output by the image encoder is as follows:

[0138] I=[I1,I2,...I j ,...I N ]

[0139] Where I is the image feature vector in one-dimensional vector data format, N is the total number of image features in the image feature vector, and I j is the jth image feature.

[0140] Step S3.4.3: The audio feature extraction subnetwork extracts the audio feature vector from the three-channel image corresponding to the audio data in the audio-text image pair.

[0141] The audio feature vector output by the audio feature extraction subnetwork is as follows:

[0142] F=[F1,F2,...F k ,...F Q ]

[0143] Where F is the audio feature vector in one-dimensional vector data format, Q is the total number of audio features in the audio feature vector, and F k is the kth audio feature.

[0144] Step S3.4.4: Calculate the cosine similarity between the text data and the image data, the cosine similarity between the audio data and the text data, and the cosine similarity between the image data and the audio data based on the normalized text feature vector, image feature vector, and audio feature vector.

[0145] The normalization method is the L2 normalization in step S1.3, which will not be described here. The normalized text feature vector T′=[T′1,T′2,…T′ i ,…T′ M ]、Image feature vector I′=[I′1,I′2,…I′ j ,…I′ N ] and audio feature vector F′=[F′1,F′2,…F′ K ,…F′ Q ]. And the cosine similarity of the three is calculated as follows:

[0146]

[0147]

[0148]

[0149] Where cosine similarities(T′,I′) is the cosine similarity between the normalized text feature vector and the image feature vector, that is, the cosine similarity between text data and image data; cosine similarities(F′,T′) is the cosine similarity between the normalized audio feature vector and the text feature vector, that is, the cosine similarity between audio data and text data; cosine similarities(I′,F′) is the cosine similarity between the normalized image feature vector and the audio feature vector, that is, the cosine similarity between image data and audio data.

[0150] Similarly, in the same audio-text-image pair, the cosine similarity between text data and image data is the same as the cosine similarity between image data and text data, the cosine similarity between audio data and text data is the same as the cosine similarity between text data and image data, and the cosine similarity between image data and audio data is the same as the cosine similarity between audio data and image data.

[0151] Step S3.4.5: Calculate the loss value based on the cosine similarity between the text data and the image data, the cosine similarity between the audio data and the text data, and the cosine similarity between the image data and the audio data, and update the cross-modal model.

[0152] like Figure 4 As shown, since the cross-modal model of this embodiment performs deep semantic understanding on the three modal data of text, image, and audio, this embodiment combines the one-dimensional feature vectors of the three data in pairs to obtain three two-dimensional feature vectors. After calculating the loss value of each two-dimensional feature vector, the loss value of the entire cross-modal model is obtained to update the cross-modal model. The loss value calculation method of each two-dimensional feature vector in this embodiment is the same as that described in step S1.4.

[0153] (1) Calculate the text image loss value based on the cosine similarity of text data and image data ti .

[0154]

[0155]

[0156]

[0157] (2) Calculate the audio text loss value based on the cosine similarity of audio data and text data ft .

[0158]

[0159]

[0160] Where, loss f is the loss value of the audio data, that is, the difference between the true audio sample label and the predicted probability, y f The true label of the audio data based on the text-audio pair, is the predicted probability value of the audio data based on the text-audio pair, that is, the cosine similarity of the audio data and the text data calculated in step S3.4.4.

[0161] (3) Calculate the image and audio loss value based on the cosine similarity of image data and audio data if .

[0162]

[0163] (4) The text image loss value, audio text loss value and image audio loss value are added and averaged as the final loss value to update the cross-modal model.

[0164]

[0165] Where loss is the final loss value, and the cross-modal model is updated based on this loss value.

[0166] Step S4: Call the ESC-50 dataset and the AudioSet dataset to optimize the audio feature extraction subnetwork in the cross-modal model after weight transfer learning to obtain a cross-modal model with optimized weights.

[0167] Because the audio data used in the above training steps is relatively small compared to the text and image data, the audio feature extraction subnetwork cannot be fully trained, resulting in low accuracy on the test set of the cross-modal deep learning algorithm model. Therefore, this example uses the ESC-50 dataset to fine-tune the audio feature extraction subnetwork, which can enhance the model's generalization performance and improve the model's test set accuracy.

[0168] During training, the audio data spectrogram in the ESC-50 dataset is first mapped to the three input channels along its frequency axis to obtain a three-channel image. Then, the audio-text pairs in the ESC-50 dataset are taken, and the weights of the text encoder and image encoder in the cross-modal model are frozen. The cross-modal model is trained with the audio-text pairs, and the weights of the audio feature extraction subnetwork are adjusted.

[0169] Then, a short-time Fourier transform (SFT) was used to obtain a spectrogram of the audio data in the AudioSet dataset. The spectrogram was mapped along its frequency axis onto three input channels, resulting in a three-channel image divided into three frequency bands: low (0.00-7.35kHz), medium (7.35-14.70kHz), and high (14.7-22.05kHz). The transformed and mapped three-channel spectrogram and the corresponding text were input into a cross-modal deep learning model with frozen image encoder parameters. The model was trained for 300 rounds using the Adam optimizer. The base learning rate was set to 0.00025, the weight decay coefficient was 0.0005, and the cosine similarity function and cross-entropy loss function were used to calculate the difference between the true value and the predicted value, that is, the audio-to-text loss value was calculated to update the text encoder and audio feature extraction subnetwork.

[0170] Step S5: Run the cross-modal model after weight optimization to output the association information between text data, image data, and audio data in the multi-source dataset, and complete the deep semantic understanding of the multi-source dataset.

[0171] Before applying the model, in order to ensure the accuracy of the model, the trained cross-modal deep learning model is used to perform an evaluation test on a multi-source dataset containing three modalities: images, audio, and text. If the test passes, the construction of the cross-modal model is completed. If the test fails, retraining is required. When retraining, step S3 can be entered for training.

[0172] The deep semantic understanding method based on the cross-modal model of this embodiment uses the text-image pairs in the WIT dataset to pre-train the text encoder and image encoder of the deep learning model with two modalities to obtain a bi-modal deep learning model after weight initialization; pre-trains the audio feature extraction subnetwork through the ImageNet dataset; uses the AudioSet dataset to perform transfer training on the audio feature extraction subnetwork, text encoder and image encoder in the cross-modal deep learning model; uses the ESC-50 dataset to fine-tune the audio feature extraction subnetwork of the cross-modal deep learning model that has undergone transfer training with two modal data (audio and text); finally, the pre-trained and fine-tuned cross-modal deep learning model is evaluated on a multi-source dataset containing three modalities of images, audio and text to complete the deep semantic understanding based on the cross-modal model.

[0173] In another embodiment, the present application also provides a deep semantic understanding device based on a cross-modal model, comprising a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the deep semantic understanding method based on a cross-modal model.

[0174] For the specific definition of a deep semantic understanding device based on a cross-modal model, please refer to the definition of a deep semantic understanding method based on a cross-modal model above, which will not be repeated here.

[0175] The memory and processor are electrically connected, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected via one or more communication buses or signal lines. The memory stores a computer program executable on the processor, and the processor implements the methods of the embodiments of the present invention by executing the computer program stored in the memory.

[0176] The memory may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a program, and the processor executes the program after receiving an execution instruction.

[0177] The processor may be an integrated circuit chip with data processing capabilities. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor.

[0178] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A deep semantic understanding method based on a cross-modal model, characterized by: The cross-modal model includes a text encoder, an image encoder, and an audio feature extraction sub-network. The deep semantic understanding method based on the cross-modal model includes: Call the WIT dataset to pre-train the text encoder and image encoder; Use the ImageNet dataset to pre-train the audio feature extraction sub-network; The AudioSet dataset is used to perform transfer training on the pre-trained text encoder, image encoder, and audio feature extraction sub-networks to obtain a cross-modal model after weight transfer learning. The ESC-50 and AudioSet datasets are used to optimize the audio feature extraction subnetwork in the cross-modal model after weight transfer learning to obtain a cross-modal model with optimized weights. Run the weight-optimized cross-modal model to output the association information between text data, image data, and audio data in the multi-source dataset, and complete the deep semantic understanding of the multi-source dataset; The calling of the AudioSet dataset to perform migration training on the pre-trained text encoder, image encoder, and audio feature extraction sub-network includes: Convert the audio data in the AudioSet dataset into a spectrogram through short-time Fourier transform; Mapping the spectrogram onto three input channels according to frequency bands to obtain a three-channel image; Take an audio-text pair from the AudioSet dataset, use the three-channel image of the audio data in the audio-text pair as input, and use the text data in the audio text as labels to perform extended training on the pre-trained audio feature extraction subnetwork; The expanded trained audio feature extraction subnetwork is combined with the pre-trained text encoder and image encoder as the cross-modal model to be trained. The audio, text, and image pairs in the AudioSet dataset are used for transfer training of the cross-modal model. The calling of the ESC-50 dataset and the AudioSet dataset to optimize the training of the audio feature extraction subnetwork in the cross-modal model after weight transfer learning includes: Take the audio-text pairs from the ESC-50 dataset, freeze the weights of the text encoder and image encoder in the cross-modal model, train the cross-modal model with the audio-text pairs, and adjust the weights of the audio feature extraction subnetwork; Take the audio-text pairs in the AudioSet dataset, freeze the weights of the image encoder in the cross-modal model, train the cross-modal model with the audio-text pairs, and adjust the weights of the audio feature extraction subnetwork and the text encoder.

2. The cross-modal model-based deep semantic understanding method according to claim 1, characterized in that: The WIT dataset is called to pre-train the text encoder and image encoder, including: The text encoder extracts text feature vectors of text data in the WIT data set; The image encoder extracts image feature vectors of image data in the WIT data set; Calculate the cosine similarity between the text data and the image data in the text-image pair based on the normalized text feature vector and the image feature vector; The loss value of the text data and the loss value of the image data are calculated based on the cosine similarity, and the text encoder and the image encoder are updated.

3. The cross-modal model-based deep semantic understanding method according to claim 2, characterized in that: The method of calculating the loss value of the text data and the loss value of the image data based on the cosine similarity and updating the text encoder and the image encoder includes: The loss value of the text data is calculated as follows: ； Where, is the loss value of text data, is the true label of the text data based on the text-image pair, is the predicted probability value of the text data based on the text-image pair, that is, the cosine similarity between the text data and the image data in the text-image pair; The loss value of the image data is calculated as follows: ； Where, is the loss value of the image data, is the true label of the image data based on the text-image pair, is the predicted probability value of the image data based on the text-image pair, that is, the cosine similarity between the text data and the image data in the text-image pair; The loss value of the text-image pair is calculated as follows: ； Where, is the loss value of the text-image pair; Backpropagation is performed based on the loss value of the text-image pair to update the weights of the text encoder and image encoder.

4. The cross-modal model-based deep semantic understanding method according to claim 1, wherein: The calling of the ImageNet dataset to pre-train the audio feature extraction sub-network includes: Use RandomErasing and Mixup methods to perform data enhancement on the ImageNet dataset; The audio feature extraction subnetwork is pre-trained using the data-augmented ImageNet dataset.

5. The cross-modal model-based deep semantic understanding method according to claim 1, wherein: The method of obtaining audio-text-image pairs from the AudioSet dataset and performing migration training on the cross-modal model using the audio-text-image pairs includes: The text encoder extracts a text feature vector of text data in the audio-text-image pair; The image encoder extracts an image feature vector of image data in an audio-text-image pair; The audio feature extraction subnetwork extracts audio feature vectors from the three-channel image corresponding to the audio data in the audio-text image pair; Calculate the cosine similarity between the text data and the image data, the cosine similarity between the audio data and the text data, and the cosine similarity between the image data and the audio data based on the normalized text feature vector, the image feature vector, and the audio feature vector; The loss value is calculated based on the cosine similarity between text data and image data, the cosine similarity between audio data and text data, and the cosine similarity between image data and audio data, and the cross-modal model is updated.

6. The cross-modal model-based deep semantic understanding method according to claim 5, characterized in that: The method of calculating the loss value based on the cosine similarity between the text data and the image data, the cosine similarity between the audio data and the text data, and the cosine similarity between the image data and the audio data and updating the cross-modal model includes: Calculate the text image loss value based on the cosine similarity of text data and image data; Calculate the audio-text loss value based on the cosine similarity between audio data and text data; Calculate the image audio loss value based on the cosine similarity of the image data and the audio data; The text image loss value, audio text loss value and image audio loss value are added and averaged as the final loss value to update the cross-modal model.

7. A deep semantic understanding device based on a cross-modal model, comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by the processor, the steps of the deep semantic understanding method based on the cross-modal model described in any one of claims 1 to 6 are implemented.

Images