Text mining data query method and system based on cross-modal similarity

Abstract

The invention provides a cross-modal similarity-based text mining data query method and system, relates to the technical field of data mining, comprises a cross-modal similarity-based text mining data query method and system, and belongs to the technical field of natural language processing. The method comprises the following steps: constructing a multi-level feature extraction network to respectively obtain text, image and time sequence feature vectors; feature fusion is realized by using an adaptive cross-modal contrast learning network, and a cross-modal similarity matrix is constructed; performing hierarchical screening on the target text by adopting a multi-level screening model to obtain a candidate text set; a text relation reasoning module is constructed based on a graph neural network, and text clustering and relation prediction are completed through a multi-task learning framework. According to the method, the accuracy of cross-modal text data query is improved, the adaptivity of feature fusion is enhanced, and efficient text relationship mining is realized.

Description

Technical Field

[0001] The present invention relates to data mining technology, and in particular to a method and system for querying data based on cross-modal similarity text mining. Background Art

[0002] Traditional cross-modal retrieval methods usually use simple concatenation or weighted averaging to fuse features from different modalities, which makes it difficult to effectively capture the complex associations between different modalities. Different modal data are heterogeneous, and simple feature fusion methods are prone to information loss or noise interference, thus affecting the accuracy of retrieval. For example, directly concatenating the semantic features of text and the visual features of images cannot reflect the fine-grained correspondence between text descriptions and image content.

[0003] Many multimedia data, such as videos, contain rich temporal information. Existing cross-modal retrieval methods often ignore the importance of temporal information, resulting in the inability to effectively handle retrieval tasks that contain temporal relationships. For example, if a user wants to retrieve a video clip of "a person running first and then drinking water", if the retrieval method cannot understand the temporal relationship between "running" and "drinking water", it will be difficult to return accurate results.

[0004] Existing cross-modal retrieval methods usually only return a simple ranked list, lacking structured expression and in-depth analysis of the retrieval results. Users often need to spend a lot of time filtering and organizing the required information from the returned results. For example, a user wants to retrieve graphic information related to "healthy lifestyle". If the retrieval results are just a simple list of graphics and text, it is difficult for the user to quickly understand the relationship between different healthy lifestyles and their respective characteristics. Ideally, the retrieval results should be presented in a more structured way, such as knowledge graphs or topic clustering, so that users can understand and use the retrieved information more effectively. Summary of the invention

[0005] The embodiments of the present invention provide a data query method and system based on cross-modal similarity text mining, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a data query method based on cross-modal similarity text mining, including:

[0008] Acquire a text to be queried and a target text in a preset text library, wherein the target text includes image feature data and temporal relationship data; construct a multi-level feature extraction network, wherein the multi-level feature extraction network includes a text feature extraction module, an image feature extraction module, and a temporal feature extraction module; input the text to be queried into the text feature extraction module to obtain a text multi-dimensional feature vector, input the image feature data into the image feature extraction module to obtain an image multi-dimensional feature vector, and input the temporal relationship data into the temporal feature extraction module to obtain a temporal multi-dimensional feature vector;

[0009] Constructing an adaptive cross-modal contrastive learning network, the adaptive cross-modal contrastive learning network includes a feature fusion subnetwork and a dynamic weight adjustment module; inputting the text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector into the feature fusion subnetwork, fusing and mapping different modal features through the dynamic weight adjustment module to obtain adaptive fusion features; optimizing based on the adaptive fusion features using a contrastive learning loss function to construct a cross-modal similarity matrix;

[0010] A multi-level screening model is constructed, wherein the multi-level screening model includes a coarse-grained filtering layer and a fine-grained evaluation layer; the target text is hierarchically screened based on the cross-modal similarity matrix using the multi-level screening model to obtain a target candidate text set; a text relationship reasoning module is constructed using a graph neural network, and a text relationship graph is established based on the target candidate text set; a multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report.

[0011] Construct a multi-level feature extraction network, including:

[0012] Constructing a hierarchical attention network of a text feature extraction module, converting the query text into an initial word vector through a word vector embedding layer; constructing a self-attention matrix based on the initial word vector, wherein the self-attention matrix obtains a word-level attention score through a scaled dot product operation of a query vector, a key vector, and a value vector; normalizing the word-level attention score and performing a weighted combination with the value vector to obtain a word-level context representation; dividing the text data into a plurality of sentence units, performing a pooling operation on the word-level features in each sentence unit to obtain a sentence representation; constructing an inter-sentence attention network to calculate the semantic dependency between sentences, and obtaining a multi-dimensional feature vector of the text;

[0013] Constructing a spatial attention network of an image feature extraction module, extracting a multi-scale feature map of the image feature data through a multi-layer convolutional network; constructing a channel attention branch and a spatial attention branch for the multi-scale feature map; in the channel attention branch, obtaining a channel descriptor through global average pooling and maximum pooling operations, and inputting the channel descriptor into a multi-layer perceptron to generate a channel weight; in the spatial attention branch, generating a two-dimensional attention map through a convolution operation; performing feature recalibration on the channel weight and the two-dimensional attention map to obtain a multi-dimensional feature vector of the image;

[0014] A bidirectional long short-term memory network of a temporal feature extraction module is constructed to map the timestamp information of the temporal relationship data into a time embedding vector; a forward long short-term memory network layer and a backward long short-term memory network layer are constructed, and both the forward long short-term memory network layer and the backward long short-term memory network layer include an input gating unit, a forgetting gating unit and an output gating unit; the input gating unit is used to control the current moment information input ratio, the forgetting gating unit is used to adjust the degree of historical information retention, and the output gating unit is used to determine the state output information amount; a temporal attention layer is constructed on top of the long short-term memory network layer, the correlation strength of different time steps is calculated, and the historical states are weightedly combined according to the attention weights to obtain a multi-dimensional feature vector of the image.

[0015] Construct an adaptive cross-modal contrastive learning network, including:

[0016] Constructing a feature fusion subnetwork, dividing the feature fusion subnetwork into a text feature transformation branch, an image feature transformation branch, and a temporal feature transformation branch; setting a projection layer in each feature transformation branch, wherein the projection layer includes a fully connected layer for feature dimension transformation, a batch normalization layer for feature distribution standardization, and an activation function for nonlinear mapping; setting a cross-modal attention layer after the projection layer, wherein the cross-modal attention layer includes a query mapping unit, a key-value mapping unit, and an attention calculation unit;

[0017] The input text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector are respectively input into the projection layers of the text feature transformation branch, the image feature transformation branch and the time series feature transformation branch; the projection layer maps the feature vectors of different dimensions to the feature space of the same dimension through the fully connected layer, the batch normalization layer and the activation function to obtain a unified feature vector; the cross-modal attention layer uses the unified feature vector of each modality as a query vector and the unified feature vectors of other modalities as a key-value pair to calculate the inter-modal attention weights, and obtains the attention-enhanced modal features;

[0018] A dynamic weight adjustment module is constructed, and the dynamic weight adjustment module includes a feature quality assessment network and a weight generation network; the feature quality assessment network receives the statistics and distribution information of the attention-enhanced modal features, and outputs a feature reliability score; the weight generation network generates dynamic fusion weights based on the feature reliability score and the attention-enhanced modal features through a multi-layer perceptron, and normalizes the dynamic fusion weights; the dynamic fusion weights are weightedly combined with the attention-enhanced modal features to obtain an adaptive fusion feature.

[0019] The contrastive learning loss function is used to optimize based on the adaptive fusion features to construct a cross-modal similarity matrix, including:

[0020] Performing data enhancement processing on the adaptive fusion features to generate positive sample pairs, and randomly sampling from the adaptive fusion features of the same batch to obtain negative sample pairs; constructing a single modality contrast loss based on the positive sample pairs and the negative sample pairs; constructing a modality alignment loss and a modality contrast loss, wherein the modality alignment loss realizes feature alignment by minimizing the feature distance of the same adaptive fusion feature in different modalities, and the modality contrast loss realizes feature discrimination by increasing the feature distance between different adaptive fusion features, thereby obtaining an optimized discrimination feature;

[0021] Based on the spatial local density of the optimized discriminant feature, a dynamic neighborhood is constructed for each query sample; in the dynamic neighborhood, the correlation weight between the sample and other samples in the neighborhood is calculated by the attention mechanism to obtain the neighborhood attention weight; and the cosine similarity of the sample pair in the optimized discriminant feature space is calculated as the initial similarity;

[0022] A graph attention network is constructed, and the neighborhood attention weights and the initial similarities are input into the graph attention network for feature propagation and aggregation to obtain a similarity representation that integrates local structural information; the similarity representation is iteratively updated to obtain a final similarity, and a cross-modal similarity matrix is ​​constructed based on the final similarities of all sample pairs, wherein the constructed cross-modal similarity matrix includes direct similarity relationships and indirect similarity information based on local structures.

[0023] Construct a graph attention network, including:

[0024] The input samples are constructed into a graph structure, in which nodes represent sample features and edges represent the strength of association between samples; an adjacency matrix of the graph is constructed based on pre-acquired neighborhood attention weights, and each node is connected to nodes in its dynamic neighborhood; the node features are mapped to multiple semantic subspaces through a multi-head attention mechanism, and each semantic subspace is used to independently learn feature association patterns;

[0025] For the node pairs in the graph structure, the node features are input into the weight matrix for linear transformation to obtain a query vector and a key vector, wherein the query vector is used to represent the feature query information of the target node, and the key vector is used to represent the feature response information of the source node; the query vector and the key vector are matrix multiplied to obtain the original attention score; the original attention score is element-wise multiplied with the pre-calculated neighborhood attention weight to obtain the local structure constraint score; the local structure constraint score is nonlinearly transformed through an activation function with a negative slope, and normalized using a function to obtain the attention coefficient between the node pairs;

[0026] A weighted matrix is ​​constructed based on the attention coefficient, and matrix multiplication operation is performed on the weighted matrix and the neighborhood features of the node to obtain aggregated features; the aggregated features are spliced ​​with the original features of the node in the feature dimension, and nonlinear feature transformation is performed through a multi-layer perceptron to obtain preliminary updated features; the preliminary updated features are residually connected with the original features of the node, and the feature distribution is adjusted through a layer normalization network to obtain standardized features; the standardized features are selectively retained through an adaptive gating mechanism to obtain the effective information of the original features and the updated features, so as to obtain a similarity representation that integrates local structural information.

[0027] The target text is hierarchically screened by using the multi-level screening model based on the cross-modal similarity matrix to obtain a target candidate text set, including:

[0028] Performing bidirectional normalization processing on the cross-modal similarity matrix, wherein the bidirectional normalization processing includes normalizing the row vectors and applying a normalization function to the column vectors to obtain a preliminary normalized similarity matrix; calculating K average similarities between each text and its neighbors as local similarity features, and performing weighted adjustment on the preliminary normalized similarity matrix based on the local similarity features to obtain a similarity matrix that strengthens the local structure;

[0029] The similarity matrix of the enhanced local structure is mapped to a low-dimensional space and subjected to product quantization discretization processing to construct a multi-layer index tree structure; an adaptive search radius is set based on the multi-layer index tree structure, an approximate nearest neighbor search is performed within the adaptive search radius, and search path information is recorded to obtain a preliminary screening candidate text; features are extracted from the preliminary screening candidate text at multiple semantic levels and feature similarity scores are calculated, and the feature similarity scores are merged with the search path information to obtain a multi-dimensional evaluation index; a weighted ranking model is constructed based on the multi-dimensional evaluation index, and the preliminary screening candidate text is re-ranked to obtain a re-ranked candidate text;

[0030] The similarity distribution characteristics of the reordered candidate texts are counted, and the screening threshold is adaptively adjusted based on the similarity distribution characteristics; the similarity between the reordered candidate texts is calculated, and the diversity constraints are performed on the reordered candidate texts based on the maximum marginal correlation algorithm to obtain an intermediate candidate text set; the confidence score of each candidate text in the intermediate candidate text set is calculated based on the feature consistency of multiple modalities; the candidate texts with confidence scores higher than a preset threshold are preferentially retained to obtain a target candidate text set.

[0031] A multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report, including:

[0032] Perform word-level feature extraction, sentence-level feature extraction and document-level feature extraction on the text node to obtain word-level features, sentence-level features and document-level features; perform weighted fusion on the word-level features, sentence-level features and document-level features to obtain text node features; based on the text node features, perform feature extraction on the relationship edge using bidirectional feature propagation to obtain initial relationship edge features; calculate information aggregation weights based on the initial relationship edge features, perform weighted aggregation on the initial relationship edge features, and obtain optimized relationship edge features;

[0033] The text node features and the optimized relationship edge features are input into a multi-task learning framework, which includes a text clustering subtask and a relationship prediction subtask; for the text clustering subtask, a node similarity matrix is ​​calculated based on the text node features; for the relationship prediction subtask, an edge feature vector is calculated based on the optimized relationship edge features; based on the node similarity matrix, a probability distribution from a sample to a cluster center is calculated, and a distance metric function is constructed in combination with the graph structure information of the text relationship graph; the text nodes are clustered according to the distance metric function and the probability distribution to obtain a text clustering result;

[0034] Based on the edge feature vector, a relation feature representation is constructed from the semantic dimension, the structural dimension and the temporal dimension, and the relation feature representation is fused using a multi-head attention mechanism; a relation prediction result is generated according to the fused relation feature representation; the text clustering result is organized into a topic hierarchy, and the relation prediction result is used as an entity relationship structure; and a structured query report is generated based on the topic hierarchy and the entity relationship structure.

[0035] According to a second aspect of the embodiments of the present invention,

[0036] Provides a data query system based on cross-modal similarity text mining, including:

[0037] The first unit is used to obtain a text to be queried and a target text in a preset text library, wherein the target text includes image feature data and temporal relationship data; construct a multi-level feature extraction network, wherein the multi-level feature extraction network includes a text feature extraction module, an image feature extraction module, and a temporal feature extraction module; input the text to be queried into the text feature extraction module to obtain a text multi-dimensional feature vector, input the image feature data into the image feature extraction module to obtain an image multi-dimensional feature vector, and input the temporal relationship data into the temporal feature extraction module to obtain a temporal multi-dimensional feature vector;

[0038] The second unit is used to construct an adaptive cross-modal contrastive learning network, which includes a feature fusion subnetwork and a dynamic weight adjustment module; the text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector are input into the feature fusion subnetwork, and different modal features are fused and mapped through the dynamic weight adjustment module to obtain adaptive fusion features; and the contrastive learning loss function is used to optimize based on the adaptive fusion features to construct a cross-modal similarity matrix;

[0039] The third unit is used to construct a multi-level screening model, which includes a coarse-grained filtering layer and a fine-grained evaluation layer; using the multi-level screening model, the target text is hierarchically screened based on the cross-modal similarity matrix to obtain a target candidate text set; using a graph neural network to build a text relationship reasoning module, and establishing a text relationship graph based on the target candidate text set; using a multi-task learning framework to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report.

[0040] According to a third aspect of the embodiments of the present invention,

[0041] An electronic device is provided, comprising:

[0042] processor;

[0043] a memory for storing processor-executable instructions;

[0044] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0045] A fourth aspect of the embodiments of the present invention is:

[0046] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0047] The beneficial effects of this application are as follows:

[0048] The present invention combines text, image and time-series relationship data to construct a cross-modal similarity matrix, and uses a multi-level screening model for hierarchical screening, effectively filtering out irrelevant information, improving query accuracy and efficiency, and being able to more accurately find target texts related to the query text. The introduction of an adaptive cross-modal contrastive learning network enables the model to adaptively fuse according to the importance of different modal features through dynamic weight adjustment, further improving the accuracy of the query.

[0049] The present invention uses graph neural networks to construct a text relationship reasoning module to cluster and predict relationships among candidate texts. It can not only provide relevant target texts, but also reveal the relationships between target texts and generate structured query reports, thereby enhancing the interpretability of query results and making it easier for users to understand the logic and associations behind the query results.

[0050] The present invention combines data of three different modalities (text, image, and temporal relationship), enabling it to be applied to a wider range of query scenarios, such as event retrieval that simultaneously contains text, images, and time information, multimedia archive retrieval, etc., breaking through the limitations of traditional single-modality text queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a flow chart of a method for querying data based on cross-modal similarity text mining according to an embodiment of the present invention;

[0052] Figure 2 The figure is a schematic diagram of the structure of a cross-modal similarity text mining data query system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0055] Figure 1 Schematic diagram of the process of a method for querying data based on cross-modal similarity text mining according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0056] Acquire a text to be queried and a target text in a preset text library, wherein the target text includes image feature data and temporal relationship data; construct a multi-level feature extraction network, wherein the multi-level feature extraction network includes a text feature extraction module, an image feature extraction module, and a temporal feature extraction module; input the text to be queried into the text feature extraction module to obtain a text multi-dimensional feature vector, input the image feature data into the image feature extraction module to obtain an image multi-dimensional feature vector, and input the temporal relationship data into the temporal feature extraction module to obtain a temporal multi-dimensional feature vector;

[0057] Constructing an adaptive cross-modal contrastive learning network, the adaptive cross-modal contrastive learning network includes a feature fusion subnetwork and a dynamic weight adjustment module; inputting the text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector into the feature fusion subnetwork, fusing and mapping different modal features through the dynamic weight adjustment module to obtain adaptive fusion features; optimizing based on the adaptive fusion features using a contrastive learning loss function to construct a cross-modal similarity matrix;

[0058] A multi-level screening model is constructed, wherein the multi-level screening model includes a coarse-grained filtering layer and a fine-grained evaluation layer; the target text is hierarchically screened based on the cross-modal similarity matrix using the multi-level screening model to obtain a target candidate text set; a text relationship reasoning module is constructed using a graph neural network, and a text relationship graph is established based on the target candidate text set; a multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report.

[0059] Get the query text and the target text in the preset text library. The target text contains image feature data and time series relationship data. For example, the query text may be "a red sports car driving on the highway", and the target text in the preset text library may be a text description containing "a red Ferrari speeding on the track", corresponding vehicle image feature data (such as color histogram, edge detection features, etc.) and a timestamp sequence indicating the vehicle driving.

[0060] Construct a multi-level feature extraction network. The network includes a text feature extraction module, an image feature extraction module, and a time series feature extraction module. The text feature extraction module can use the pre-trained BERT model to convert the query text "red sports car driving on the highway" into a 768-dimensional word vector sequence, and then obtain the multi-dimensional feature vector of the text through average pooling. The image feature extraction module can use a convolutional neural network (CNN), such as ResNet50, to extract the color, texture, shape and other features of the image, and convert the image feature data into a 2048-dimensional feature vector. The time series feature extraction module can use a recurrent neural network (RNN), such as LSTM, to process the timestamp sequence, capture the time series relationship, and output a 128-dimensional time series feature vector.

[0061] Construct an adaptive cross-modal contrastive learning network. The network includes a feature fusion subnetwork and a dynamic weight adjustment module. Input the text multidimensional feature vector, the image multidimensional feature vector, and the time series multidimensional feature vector into the feature fusion subnetwork. The feature fusion subnetwork can use a multi-layer perceptron (MLP) to map the feature vectors of different modalities to the same dimensional space, such as 512 dimensions. The dynamic weight adjustment module dynamically adjusts the weights of different modal features according to their importance. For example, if the text description is highly correlated with the image content, the weights of the text and image features will be higher. The dynamic weight adjustment module can use an attention mechanism to calculate the weights according to the correlation between the features of different modalities. The weighted feature vectors are concatenated to obtain adaptive fusion features. The cross-modal similarity matrix is ​​constructed by optimizing based on the adaptive fusion features using the contrastive learning loss function. For example, the similarity matrix is ​​constructed by calculating the cosine similarity between the query text and the adaptive fusion features of each target text.

[0062] Construct a multi-level screening model. The model includes a coarse-grained filtering layer and a fine-grained evaluation layer. Using the multi-level screening model, the target text is hierarchically screened based on the cross-modal similarity matrix to obtain a set of target candidate texts. For example, the coarse-grained filtering layer sets a threshold to filter out texts with similarity below the threshold; the fine-grained evaluation layer uses a more complex sorting algorithm, such as a BPR-based sorting algorithm, to sort the remaining texts and select the top K texts as the target candidate text set. Assume that K = 10, and 10 candidate texts are obtained.

[0063] A text relationship reasoning module is constructed using a graph neural network, and a text relationship graph is established based on the target candidate text set. For example, 10 candidate texts are used as nodes of the graph, and the edges between the nodes represent the similarity between the texts. A graph convolutional network (GCN) can be used to model the text relationship graph.

[0064] A multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report. For example, text clustering and relationship prediction are performed simultaneously, and the loss functions of the two tasks are weighted summed as the final loss function. Text clustering groups similar texts, and relationship prediction predicts the relationship between texts, such as "contains" and "belongs to". Finally, a structured query report containing text clustering results and relationship prediction results is generated.

[0065] Beneficial effects:

[0066] By integrating text, images and time series information, we can more accurately understand the user's query intention, thereby improving the accuracy of the query results.

[0067] This method supports query based on multiple modalities such as text, image and time series relationship, which expands the query mode and is convenient for users.

[0068] Through the text relationship reasoning module, structured query reports can be generated to provide more comprehensive and easier to understand query results.

[0069] In an optional implementation, a multi-level feature extraction network is constructed, including:

[0070] Constructing a hierarchical attention network of a text feature extraction module, converting the query text into an initial word vector through a word vector embedding layer; constructing a self-attention matrix based on the initial word vector, wherein the self-attention matrix obtains a word-level attention score through a scaled dot product operation of a query vector, a key vector, and a value vector; normalizing the word-level attention score and performing a weighted combination with the value vector to obtain a word-level context representation; dividing the text data into a plurality of sentence units, performing a pooling operation on the word-level features in each sentence unit to obtain a sentence representation; constructing an inter-sentence attention network to calculate the semantic dependency between sentences, and obtaining a multi-dimensional feature vector of the text;

[0071] Constructing a spatial attention network of an image feature extraction module, extracting a multi-scale feature map of the image feature data through a multi-layer convolutional network; constructing a channel attention branch and a spatial attention branch for the multi-scale feature map; in the channel attention branch, obtaining a channel descriptor through global average pooling and maximum pooling operations, and inputting the channel descriptor into a multi-layer perceptron to generate a channel weight; in the spatial attention branch, generating a two-dimensional attention map through a convolution operation; performing feature recalibration on the channel weight and the two-dimensional attention map to obtain a multi-dimensional feature vector of the image;

[0072] A bidirectional long short-term memory network of a temporal feature extraction module is constructed to map the timestamp information of the temporal relationship data into a time embedding vector; a forward long short-term memory network layer and a backward long short-term memory network layer are constructed, and both the forward long short-term memory network layer and the backward long short-term memory network layer include an input gating unit, a forgetting gating unit and an output gating unit; the input gating unit is used to control the current moment information input ratio, the forgetting gating unit is used to adjust the degree of historical information retention, and the output gating unit is used to determine the state output information amount; a temporal attention layer is constructed on top of the long short-term memory network layer, the correlation strength of different time steps is calculated, and the historical states are weightedly combined according to the attention weights to obtain a multi-dimensional feature vector of the image.

[0073] In order to more effectively extract key information from multimodal data, this embodiment proposes a multi-level feature extraction network that integrates multi-dimensional features of text, image and time series data.

[0074] First, perform text feature extraction. Input the query text into the word vector embedding layer, such as using Word2Vec or the Glove model, and convert each word into an initial word vector of a fixed dimension, such as 300 dimensions. Suppose the input text is "Today's weather is sunny, suitable for outdoor sports." "Today", "weather", "sunny", "suitable", "outdoor", and "sports" will be converted into corresponding word vectors respectively.

[0075] Then, a self-attention matrix is ​​constructed based on the initial word vector. The word vector of "today" is used as the query vector, and dot product operations are performed with the word vectors of "today", "weather", "sunny", "suitable", "outdoors", and "sports" (as key vectors and value vectors) to obtain word-level attention scores. This operation is performed on all words to obtain a self-attention matrix. In order to avoid the gradient vanishing problem, the attention score is scaled by dividing it by the square root of the word vector dimension.

[0076] Next, the word-level attention scores are normalized, for example, using a normalized exponential function, so that the sum of the attention scores of each word is 1. The normalized attention scores are then weighted and combined with the corresponding value vectors to obtain the word-level context representation of each word. For example, the word-level context representation of "today" is the weighted sum of the word vectors of all other words and their corresponding attention scores.

[0077] The text data is divided into multiple sentence units according to punctuation marks, such as the two sentences "Today's weather is fine" and "Suitable for outdoor sports". The word-level features in each sentence unit are pooled, such as average pooling or maximum pooling, to obtain the sentence representation of each sentence.

[0078] Furthermore, an inter-sentence attention network is constructed to calculate the semantic dependencies between sentences. The sentence representation of each sentence is taken as input, the attention score between sentences is calculated, and then the attention score is weightedly combined with the sentence representation to obtain the final multi-dimensional feature vector of the text.

[0079] Secondly, perform image feature extraction. Input the image data into a multi-layer convolutional network, such as a residual network or a VGG network, to extract multi-scale feature maps. Assuming that the input image is a 224x224x3 color image, after passing through the convolutional network, multiple feature maps of different scales are obtained, such as 112x112x64, 56x56x128, 28x28x256, etc.

[0080] For each scale feature map, a channel attention branch and a spatial attention branch are constructed. In the channel attention branch, global average pooling and maximum pooling operations are performed on the feature map to obtain two channel descriptors. These two channel descriptors are concatenated and input into the multi-layer perceptron to generate channel weights.

[0081] In the spatial attention branch, a convolution operation is performed on the feature map to generate a two-dimensional attention map.

[0082] Multiply the channel weight with the two-dimensional attention map, recalibrate the feature map, and obtain the multi-dimensional feature vector of the image.

[0083] Finally, perform time series feature extraction. Map the timestamp information of the time series relationship data into a time embedding vector. For example, use sine and cosine functions to encode the timestamp into a vector of fixed dimension. Assuming the timestamps are 1, 2, 3, 4, 5, convert each timestamp into a 128-dimensional time embedding vector.

[0084] Construct a forward long short-term memory network layer and a backward long short-term memory network layer. Both network layers contain input gating units, forget gating units, and output gating units. The input gating unit controls the current information input ratio, the forget gating unit adjusts the degree of historical information retention, and the output gating unit determines the amount of state output information.

[0085] Taking timestamp 1 as an example, the forward LSTM receives the embedding vector of timestamp 1 as input, and the backward LSTM receives the embedding vector of timestamp 5 as input. The flow of information is controlled by the input gate, forget gate, and output gate to obtain the forward and backward hidden states respectively.

[0086] The hidden states of the forward and backward LSTMs are concatenated to obtain the complete state representation at each time step.

[0087] A temporal attention layer is built on top of the LSTM layer to calculate the association strength of different time steps. The historical states are weighted and combined according to the attention weights to obtain the final temporal multi-dimensional feature vector.

[0088] The beneficial effects are reflected in three aspects:

[0089] Through the multi-level attention mechanism, the network can more effectively capture key information in text, images and time series data, and improve feature expression capabilities.

[0090] The attention mechanism can highlight the contribution of important features, enhance the interpretability of the model, and help understand the decision-making process of the model.

[0091] By fusing multimodal information, the model can better cope with complex scenarios and improve generalization performance.

[0092] In an optional implementation, constructing an adaptive cross-modal contrastive learning network includes:

[0093] Constructing a feature fusion subnetwork, dividing the feature fusion subnetwork into a text feature transformation branch, an image feature transformation branch, and a temporal feature transformation branch; setting a projection layer in each feature transformation branch, wherein the projection layer includes a fully connected layer for feature dimension transformation, a batch normalization layer for feature distribution standardization, and an activation function for nonlinear mapping; setting a cross-modal attention layer after the projection layer, wherein the cross-modal attention layer includes a query mapping unit, a key-value mapping unit, and an attention calculation unit;

[0094] The input text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector are respectively input into the projection layers of the text feature transformation branch, the image feature transformation branch and the time series feature transformation branch; the projection layer maps the feature vectors of different dimensions to the feature space of the same dimension through the fully connected layer, the batch normalization layer and the activation function to obtain a unified feature vector; the cross-modal attention layer uses the unified feature vector of each modality as a query vector and the unified feature vectors of other modalities as a key-value pair to calculate the inter-modal attention weights, and obtains the attention-enhanced modal features;

[0095] A dynamic weight adjustment module is constructed, and the dynamic weight adjustment module includes a feature quality assessment network and a weight generation network; the feature quality assessment network receives the statistics and distribution information of the attention-enhanced modal features, and outputs a feature reliability score; the weight generation network generates dynamic fusion weights based on the feature reliability score and the attention-enhanced modal features through a multi-layer perceptron, and normalizes the dynamic fusion weights; the dynamic fusion weights are weightedly combined with the attention-enhanced modal features to obtain an adaptive fusion feature.

[0096] In order to realize adaptive cross-modal contrastive learning, the present invention proposes a new network structure, which is mainly composed of a feature fusion subnetwork and a dynamic weight adjustment module.

[0097] First, a feature fusion subnetwork is constructed. This subnetwork is divided into a text feature transformation branch, an image feature transformation branch, and a temporal feature transformation branch, and the three branches have similar structures. Taking the text feature transformation branch as an example, it contains a projection layer and a cross-modal attention layer. The projection layer consists of a fully connected layer, a batch normalization layer, and an activation function. The input text multidimensional feature vector, such as a 768-dimensional vector, first undergoes a dimension transformation through a fully connected layer, assuming that it is transformed to 256 dimensions. Then, the batch normalization layer normalizes the transformed features to make their mean zero and variance one. Finally, the activation function, such as a linear rectification function, performs nonlinear mapping on the standardized features to enhance the expressive power of the model. The other two branches also perform similar processing to map the image features and temporal features to the same 256-dimensional feature space.

[0098] After being processed by the projection layer, the three branches obtain a unified feature vector of 256 dimensions respectively. Next, the cross-modal attention layer processes these feature vectors. Taking text features as an example, text features are used as query vectors, and image features and time series features are used as key-value pairs. The attention calculation unit calculates the attention weights between text features and image features, and between text features and time series features. For example, the scaled dot product attention mechanism can be used to calculate the dot product between the query vector and each key vector, and then the dot product is converted into a probability distribution, that is, the attention weight, through a normalized exponential function. Finally, the value vector is weighted and summed using the attention weight to obtain the attention-enhanced text features. The other two branches are also processed similarly to obtain the attention-enhanced image features and time series features.

[0099] Then, a dynamic weight adjustment module is constructed. This module includes a feature quality assessment network and a weight generation network. The feature quality assessment network receives the statistics and distribution information of the attention-enhanced modal features, such as mean, variance, kurtosis, etc., and outputs a feature reliability score. For example, if the feature distribution of a modality is relatively concentrated and the variance is small, its reliability score may be high. The weight generation network generates dynamic fusion weights through a multi-layer perceptron based on the feature reliability score and the attention-enhanced modal features. For example, the feature reliability score and the attention-enhanced modal features are spliced ​​together as the input of the multi-layer perceptron, and the weights corresponding to the three modalities are output. Finally, the dynamic fusion weights are normalized, for example, using a normalized exponential function to ensure that the sum of the three weights is 1.

[0100] Finally, the dynamic fusion weights are weighted and combined with the attention-enhanced modal features to obtain the adaptive fusion features. For example, assuming that the dynamic fusion weights of text, image, and time series features are 0.2, 0.5, and 0.3, respectively, the attention-enhanced text features are multiplied by 0.2, the attention-enhanced image features are multiplied by 0.5, and the attention-enhanced time series features are multiplied by 0.3, and then the three results are added together to obtain the final adaptive fusion features.

[0101] For example, input a text describing a cat playing, a picture of a cat playing, and a video clip of a cat playing. The text feature vector dimension is 768, the image feature vector dimension is 1024, and the time series feature vector dimension is 512. After being processed by the feature fusion sub-network, the feature vectors of the three modalities are mapped to a 256-dimensional space and enhanced by the cross-modal attention mechanism. Assume that the feature quality assessment network determines that the image modality has the highest quality, the text modality is second, and the time series modality has the lowest quality. Then the weight generation network may generate weight vectors such as 0.2, 0.6, and 0.2. In the end, the fused feature vector will be more affected by the image modality.

[0102] The beneficial effects provided by the invention can be summarized into the following three aspects:

[0103] Through the cross-modal attention mechanism, features from different modalities can complement and enhance each other, resulting in a more comprehensive and expressive feature representation.

[0104] The dynamic weight adjustment module can dynamically adjust the fusion weights according to the quality of different modal features, avoiding the limitations of the fixed weight fusion method and making better use of multimodal information.

[0105] By using a feature quality assessment network to evaluate the quality of features of different modalities, the impact of noise and low-quality features can be effectively reduced and the robustness of the model can be enhanced.

[0106] In an optional implementation, a contrastive learning loss function is used to optimize based on adaptive fusion features to construct a cross-modal similarity matrix, including:

[0107] Performing data enhancement processing on the adaptive fusion features to generate positive sample pairs, and randomly sampling from the adaptive fusion features of the same batch to obtain negative sample pairs; constructing a single modality contrast loss based on the positive sample pairs and the negative sample pairs; constructing a modality alignment loss and a modality contrast loss, wherein the modality alignment loss realizes feature alignment by minimizing the feature distance of the same adaptive fusion feature in different modalities, and the modality contrast loss realizes feature discrimination by increasing the feature distance between different adaptive fusion features, thereby obtaining an optimized discrimination feature;

[0108] Based on the spatial local density of the optimized discriminant feature, a dynamic neighborhood is constructed for each query sample; in the dynamic neighborhood, the correlation weight between the sample and other samples in the neighborhood is calculated by the attention mechanism to obtain the neighborhood attention weight; and the cosine similarity of the sample pair in the optimized discriminant feature space is calculated as the initial similarity;

[0109] A graph attention network is constructed, and the neighborhood attention weights and the initial similarities are input into the graph attention network for feature propagation and aggregation to obtain a similarity representation that integrates local structural information; the similarity representation is iteratively updated to obtain a final similarity, and a cross-modal similarity matrix is ​​constructed based on the final similarities of all sample pairs, wherein the constructed cross-modal similarity matrix includes direct similarity relationships and indirect similarity information based on local structures.

[0110] In order to construct a cross-modal similarity matrix, this embodiment proposes a solution based on contrastive learning loss function and adaptive fusion feature optimization. This solution uses contrastive learning to optimize feature representation and combines graph attention network to fuse local structure information, thereby improving the performance of cross-modal retrieval.

[0111] First, extract data features of different modalities and project them into a common feature space to obtain adaptive fusion features. For example, for image and text modalities, use pre-trained convolutional neural networks and transformer models to extract features, and then project the extracted features to the same dimension, such as 512 dimensions, through a fully connected layer. Assume that a batch contains 16 samples, each of which contains data from both image and text modalities.

[0112] Next, the adaptive fusion features are data enhanced to generate positive sample pairs. Data enhancement methods may include random cropping, color jittering, adding noise, etc. For example, the image modality is randomly cropped, and the cropped image and the original image are used as positive sample pairs. Negative sample pairs are randomly sampled from other adaptive fusion features in the same batch. For example, a sample is randomly selected from the remaining 15 samples and paired with the current sample to form a negative sample pair.

[0113] Then, a unimodal contrastive loss is constructed based on the positive sample pairs and the negative sample pairs. The goal of the contrastive loss is to bring the distance between the positive sample pairs closer and push the distance between the negative sample pairs farther. In addition, modality alignment loss and modality contrastive loss are constructed. Modality alignment loss achieves feature alignment by minimizing the feature distance of the same adaptive fusion feature in different modalities. For example, the Euclidean distance between the image modality feature and the text modality feature is calculated and used as part of the loss function. Modality contrastive loss achieves feature discrimination by increasing the feature distance between different adaptive fusion features. For example, the Euclidean distance between the image modality features of different samples is calculated and used as part of the loss function. By minimizing the weighted sum of these three loss functions, the optimized discriminant features are obtained.

[0114] Next, a dynamic neighborhood is constructed for each query sample based on the spatial local density of the optimized discriminant features. For example, the Euclidean distance between each sample and other samples in the feature space is calculated, and the K samples with the closest distance are selected as its neighborhood according to the distance sorting. Assume that K = 5.

[0115] In a dynamic neighborhood, the weight of the correlation between a sample and other samples in the neighborhood is calculated through the attention mechanism to obtain the neighborhood attention weight. For example, the weight can be calculated using a multi-head self-attention mechanism.

[0116] Then, the cosine similarity of the sample pairs in the optimized discriminant feature space is calculated as the initial similarity.

[0117] A graph attention network is constructed, and the neighborhood attention weights and initial similarities are input into the graph attention network for feature propagation and aggregation to obtain a similarity representation that integrates local structural information.

[0118] The similarity representation is iteratively updated to obtain the final similarity. The iterative update method can be a multi-layer graph attention network.

[0119] Finally, a cross-modal similarity matrix is ​​constructed based on the final similarities of all sample pairs. This matrix contains direct similarity relationships and indirect similarity information based on local structures. For example, a 16x16 matrix, where each element represents the final similarity of the corresponding sample pair.

[0120] Suppose there are two samples, their image features are [0.1, 0.2, 0.3] and [0.2, 0.3, 0.4], and their text features are [0.4, 0.5, 0.6] and [0.5, 0.6, 0.7]. After the above steps, the corresponding value of these two samples in the similarity matrix is ​​0.95.

[0121] The beneficial effects of this program can be summarized in the following three aspects:

[0122] Through contrastive learning and modality alignment, more discriminative feature representations are learned, thereby improving the accuracy of cross-modal retrieval.

[0123] Through the graph attention network, local structural information is integrated to better capture the relationship between samples and further improve the retrieval performance.

[0124] Adaptive feature fusion and data augmentation strategies improve the model’s robustness to noise and data variations.

[0125] In an optional implementation, constructing a graph attention network includes:

[0126] The input samples are constructed into a graph structure, in which nodes represent sample features and edges represent the strength of association between samples; an adjacency matrix of the graph is constructed based on pre-acquired neighborhood attention weights, and each node is connected to nodes in its dynamic neighborhood; the node features are mapped to multiple semantic subspaces through a multi-head attention mechanism, and each semantic subspace is used to independently learn feature association patterns;

[0127] For the node pairs in the graph structure, the node features are input into the weight matrix for linear transformation to obtain a query vector and a key vector, wherein the query vector is used to represent the feature query information of the target node, and the key vector is used to represent the feature response information of the source node; the query vector and the key vector are matrix multiplied to obtain the original attention score; the original attention score is element-wise multiplied with the pre-calculated neighborhood attention weight to obtain the local structure constraint score; the local structure constraint score is nonlinearly transformed through an activation function with a negative slope, and normalized using a function to obtain the attention coefficient between the node pairs;

[0128] A weighted matrix is ​​constructed based on the attention coefficient, and matrix multiplication operation is performed on the weighted matrix and the neighborhood features of the node to obtain aggregated features; the aggregated features are spliced ​​with the original features of the node in the feature dimension, and nonlinear feature transformation is performed through a multi-layer perceptron to obtain preliminary updated features; the preliminary updated features are residually connected with the original features of the node, and the feature distribution is adjusted through a layer normalization network to obtain standardized features; the standardized features are selectively retained through an adaptive gating mechanism to obtain the effective information of the original features and the updated features, so as to obtain a similarity representation that integrates local structural information.

[0129] A method for constructing a graph attention network for learning similarity representation between samples is implemented as follows:

[0130] First, construct the input samples into a graph structure. The feature of each sample is represented as a node in the graph. The strength of the association between samples is represented as the edge in the graph. For example, suppose there are three samples, and their feature vectors are [0.2, 0.5, 0.8], [0.1, 0.7, 0.2], and [0.3, 0.2, 0.5]. The strength of the association between samples can be calculated based on the cosine similarity between the feature vectors. Assuming that the calculated similarities are 0.8, 0.6, and 0.7, the three samples can be constructed into a graph, where the nodes represent the sample features and the edges represent the strength of the association between samples.

[0131] Next, the adjacency matrix of the graph is constructed based on the pre-acquired neighborhood attention weights. Each node establishes a connection relationship with the nodes in its dynamic neighborhood. The neighborhood attention weight can be obtained by calculating the similarity or distance between nodes. For example, a threshold can be set, and only nodes with a similarity greater than the threshold are considered neighbor nodes. Assuming the threshold is 0.7, in the above example, sample 1 and sample 3 are neighbors, and sample 2 and sample 3 are also neighbors. In this way, an adjacency matrix can be constructed, where 1 indicates connection and 0 indicates disconnection.

[0132] Then, the node features are mapped to multiple semantic subspaces through a multi-head attention mechanism, and each semantic subspace is used to independently learn feature association patterns. For example, 8 attention heads can be used to map the feature vector of each node to 8 different semantic subspaces.

[0133] For node pairs in the graph structure, the node features are input into the weight matrix for linear transformation to obtain the query vector and key vector. The query vector is used to represent the feature query information of the target node, and the key vector is used to represent the feature response information of the source node. For example, for sample 1 and sample 3, their feature vectors can be multiplied by different weight matrices to obtain the corresponding query vector and key vector.

[0134] The query vector is matrix multiplied with the key vector to get the raw attention score. For example, the query vector of sample 1 is matrix multiplied with the key vector of sample 3 to get a score indicating how much sample 1 pays attention to sample 3.

[0135] The local structure constraint score is obtained by element-wise multiplication of the raw attention score and the pre-computed neighborhood attention weight. For example, the local structure constraint score is obtained by multiplying the raw attention scores of sample 1 and sample 3 with the neighborhood attention weight (0.7) between them.

[0136] The local structure constraint score is nonlinearly transformed by an activation function with a negative slope, and normalized by a normalized exponential function to obtain the attention coefficient between the node pairs. For example, the local structure constraint score can be nonlinearly transformed by a leaky corrected linear unit activation function, and then normalized by a normalized exponential function to obtain the attention coefficient between sample 1 and sample 3.

[0137] A weighted matrix is ​​constructed based on the attention coefficient, and the weighted matrix is ​​multiplied with the neighborhood features of the node to obtain the aggregated features. For example, the feature vector of sample 3 is multiplied by the attention coefficient between samples 1 and 3, and then the results are added to obtain the aggregated features of sample 3.

[0138] The aggregated features are concatenated with the original features of the nodes in the feature dimension, and nonlinear feature transformation is performed through a multi-layer perceptron to obtain preliminary updated features. For example, the aggregated features of sample 3 are concatenated with its original features, and then nonlinear transformation is performed through a two-layer multi-layer perceptron to obtain preliminary updated features.

[0139] The initial update features are residually connected to the original features of the nodes, and the feature distribution is adjusted through the layer normalization network to obtain the standardized features. For example, the initial update features are added to the original features of sample 3, and then adjusted through the layer normalization network to obtain the standardized features.

[0140] The standardized features are selectively retained through an adaptive gating mechanism to retain the effective information of the original features and the updated features, and a similarity representation that integrates the local structural information is obtained. For example, the si gmo id function can be used to control the retention ratio of the original features and the updated features, and finally a similarity representation that integrates the local structural information is obtained.

[0141] The beneficial effects of this method can be summarized in the following three aspects:

[0142] By constructing samples into a graph structure and using the graph attention network to learn the association between nodes, the complex relationship between samples can be better captured, thereby improving the expressiveness of the model.

[0143] The attention mechanism can help us understand which nodes and features the model focuses on, thereby enhancing the interpretability of the model.

[0144] By fusing local structural information, we can better learn the similarity representation of samples, thereby improving the performance of the model in various downstream tasks, such as classification and clustering.

[0145] In an optional implementation, the multi-level screening model is used to perform hierarchical screening on the target text based on the cross-modal similarity matrix to obtain a target candidate text set, including:

[0146] Performing bidirectional normalization processing on the cross-modal similarity matrix, wherein the bidirectional normalization processing includes normalizing the row vectors and applying a normalization function to the column vectors to obtain a preliminary normalized similarity matrix; calculating K average similarities between each text and its neighbors as local similarity features, and performing weighted adjustment on the preliminary normalized similarity matrix based on the local similarity features to obtain a similarity matrix that strengthens the local structure;

[0147] The similarity matrix of the enhanced local structure is mapped to a low-dimensional space and subjected to product quantization discretization processing to construct a multi-layer index tree structure; an adaptive search radius is set based on the multi-layer index tree structure, an approximate nearest neighbor search is performed within the adaptive search radius, and search path information is recorded to obtain a preliminary screening candidate text; features are extracted from the preliminary screening candidate text at multiple semantic levels and feature similarity scores are calculated, and the feature similarity scores are merged with the search path information to obtain a multi-dimensional evaluation index; a weighted ranking model is constructed based on the multi-dimensional evaluation index, and the preliminary screening candidate text is re-ranked to obtain a re-ranked candidate text;

[0148] The similarity distribution characteristics of the reordered candidate texts are counted, and the screening threshold is adaptively adjusted based on the similarity distribution characteristics; the similarity between the reordered candidate texts is calculated, and the diversity constraints are performed on the reordered candidate texts based on the maximum marginal correlation algorithm to obtain an intermediate candidate text set; the confidence score of each candidate text in the intermediate candidate text set is calculated based on the feature consistency of multiple modalities; the candidate texts with confidence scores higher than a preset threshold are preferentially retained to obtain a target candidate text set.

[0149] A cross-modal text retrieval method aims to quickly and accurately find candidate texts that are semantically similar to the target text from a large amount of text data. This method uses a multi-level screening model to perform a series of processing and screening on the cross-modal similarity matrix, and finally obtains a high-quality target candidate text set.

[0150] First, the input cross-modal similarity matrix is ​​bidirectionally normalized. This step includes normalizing each row of the matrix so that the sum of the values ​​in each row is 1, and applying a normalization function to each column of the matrix, such as the commonly used normalized exponential function, to convert the values ​​in each column into a probability distribution.

[0151] For example, a 3x3 similarity matrix:

[0152] [[0.1,0.2,0.3],[0.4,0.5,0.6],[0.7,0.8,0.9]], after row normalization becomes [[0.14,0.29,0.43],[0.27,0.33,0.40],[0.26,0.30,0.34]],

[0153] After column normalization (such as normalized exponential function), it becomes:

[0154] [[0.24,0.24,0.26],[0.33,0.33,0.34],[0.43,0.43,0.40]]. The preliminary normalized similarity matrix is ​​obtained.

[0155] Next, calculate the average similarity of each text with its K nearest neighbors as the local similarity feature. Assuming K = 2, for the first text, its nearest neighbors are the second and third texts, then its local similarity feature is (0.24 + 0.26) / 2 = 0.25. Similarly, calculate the local similarity features of all texts. Then, based on these local similarity features, perform a weighted adjustment on the preliminary normalized similarity matrix. For example, multiply the local similarity feature of each text with its corresponding row vector to obtain a similarity matrix that strengthens the local structure. Taking the first text as an example, its corresponding row vector [0.24, 0.24, 0.26] is multiplied by its local similarity feature 0.25 to obtain [0.06, 0.06, 0.065].

[0156] Then, the similarity matrix of the enhanced local structure is mapped to a low-dimensional space, for example, using a dimensionality reduction method such as PCA or t-SNE. The data in the low-dimensional space is discretized by product quantization, for example, using a K-means clustering algorithm. Based on these discretized data points, a multi-layer index tree structure is constructed, such as a KD tree or a Ball tree.

[0157] On the constructed multi-layer index tree structure, set an adaptive search radius. For example, dynamically adjust the search radius based on the location of the target text in the low-dimensional space and the distribution of the data. Perform an approximate nearest neighbor search within the adaptive search radius and record the search path information, such as the tree nodes passed and the distance to each node. Obtain the initial screening candidate text.

[0158] Extract features from the initial candidate texts at multiple semantic levels, such as word vectors, sentence vectors, and paragraph vectors. Calculate similarity scores of these features, such as using cosine similarity. Fuse these feature similarity scores with the search path information, such as weighted summing them, to obtain multi-dimensional evaluation indicators.

[0159] A weighted ranking model is constructed based on multi-dimensional evaluation indicators, such as using linear regression or learning to rank algorithms. The initially screened candidate texts are re-ranked to obtain re-ranked candidate texts.

[0160] Statistically reorder the similarity distribution characteristics of the candidate texts, such as the mean and variance of the similarity. Adaptively adjust the screening threshold based on these distribution characteristics. For example, dynamically adjust the size of the screening threshold based on the mean and variance of the similarity.

[0161] Calculate the similarity between the re-ranking candidate texts, for example, using cosine similarity. Apply diversity constraints to the re-ranking candidate texts based on the maximum marginal correlation algorithm, for example, select texts with lower similarity but still relevant to the target text. Obtain a set of intermediate candidate texts.

[0162] The confidence score of each candidate text in the intermediate candidate text set is calculated based on the feature consistency of multiple modalities. For example, the feature similarities of different modalities are weighted averaged to obtain the final confidence score.

[0163] The candidate texts with confidence scores higher than a preset threshold are retained first to obtain a target candidate text set.

[0164] The beneficial effects of this method can be summarized in the following three aspects:

[0165] Through the multi-layer index tree structure and adaptive search radius, candidate texts can be quickly screened out from massive text data, significantly improving retrieval efficiency.

[0166] Through multi-level screening models and multi-dimensional evaluation indicators, the similarity between candidate texts and target texts can be evaluated more accurately, thereby improving retrieval accuracy.

[0167] By using the maximum marginal correlation algorithm to constrain diversity, we can avoid returning candidate texts that are too similar, ensure the diversity of results, and thus meet the diverse needs of users.

[0168] In an optional implementation, a multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report, including:

[0169] Perform word-level feature extraction, sentence-level feature extraction and document-level feature extraction on the text node to obtain word-level features, sentence-level features and document-level features; perform weighted fusion on the word-level features, sentence-level features and document-level features to obtain text node features; based on the text node features, perform feature extraction on the relationship edge using bidirectional feature propagation to obtain initial relationship edge features; calculate information aggregation weights based on the initial relationship edge features, perform weighted aggregation on the initial relationship edge features, and obtain optimized relationship edge features;

[0170] The text node features and the optimized relationship edge features are input into a multi-task learning framework, which includes a text clustering subtask and a relationship prediction subtask; for the text clustering subtask, a node similarity matrix is ​​calculated based on the text node features; for the relationship prediction subtask, an edge feature vector is calculated based on the optimized relationship edge features; based on the node similarity matrix, a probability distribution from a sample to a cluster center is calculated, and a distance metric function is constructed in combination with the graph structure information of the text relationship graph; the text nodes are clustered according to the distance metric function and the probability distribution to obtain a text clustering result;

[0171] Based on the edge feature vector, a relation feature representation is constructed from the semantic dimension, the structural dimension and the temporal dimension, and the relation feature representation is fused using a multi-head attention mechanism; a relation prediction result is generated according to the fused relation feature representation; the text clustering result is organized into a topic hierarchy, and the relation prediction result is used as an entity relationship structure; and a structured query report is generated based on the topic hierarchy and the entity relationship structure.

[0172] In order to achieve effective analysis and query of text relationship graphs, a text clustering and relationship prediction method based on a multi-task learning framework was proposed to generate a structured query report.

[0173] First, feature extraction is performed on text nodes. Take a text relationship graph containing multiple documents as an example, where the document content includes "artificial intelligence is developing rapidly", "deep learning is the key technology of artificial intelligence", "machine learning is the foundation of artificial intelligence", etc. Each document is preprocessed by word segmentation, part-of-speech tagging, named entity recognition, etc. At the word level, a word embedding model (such as Word2Vec, Glove) can be used to convert each word into a vector representation as a word-level feature. At the sentence level, a sentence embedding model (such as Sentence-BERT) can be used to convert each sentence into a vector representation as a sentence-level feature. At the document level, the word frequency, TF-IDF value, etc. in the document can be counted, or a model such as Doc2Vec can be used to convert the document into a vector representation as a document-level feature. Then, the word-level features, sentence-level features, and document-level features are weighted and fused. For example, weights can be set based on experience or experimental results, and the three feature vectors can be linearly combined to obtain the final text node features.

[0174] Next, feature extraction is performed on the relationship edge. Using the text node features obtained in the previous step, a bidirectional feature propagation algorithm is used to extract features from the relationship edge. Specifically, for the relationship edge connecting node A and node B, the feature vector of node A and the feature vector of node B are concatenated or otherwise combined to obtain the initial relationship edge feature. Then, the information aggregation weight is calculated based on the initial relationship edge feature. For example, the weight can be calculated based on the cosine similarity of the feature vectors of the two nodes. Finally, the initial relationship edge features are weighted aggregated to obtain the optimized relationship edge features.

[0175] Then, the text node features and the optimized relationship edge features are input into the multi-task learning framework. The framework includes a text clustering subtask and a relationship prediction subtask. For the text clustering subtask, the node similarity matrix is ​​calculated based on the text node features. For example, cosine similarity or Euclidean distance can be used to calculate the similarity between nodes. For the relationship prediction subtask, the edge feature vector is calculated based on the optimized relationship edge features.

[0176] In the text clustering subtask, the probability distribution of samples to cluster centers is calculated based on the node similarity matrix. At the same time, the distance metric function is constructed in combination with the graph structure information of the text relationship graph. For example, the shortest path length between nodes can be included in the distance metric function. The text nodes are clustered according to the distance metric function and the probability distribution, for example, using the K-Means algorithm to obtain the text clustering results. For example, documents such as "artificial intelligence is developing rapidly", "deep learning is the key technology of artificial intelligence", and "machine learning is the foundation of artificial intelligence" are clustered under the theme of "artificial intelligence".

[0177] In the relationship prediction subtask, the relationship feature representation is constructed from the semantic dimension, structural dimension and time series dimension based on the edge feature vector. For example, the semantic dimension can be represented by the word vector similarity of the two nodes connected by the edge; the structural dimension can be represented by the connection relationship between the two nodes in the graph; and the time series dimension can be represented by the time order of the two nodes. A multi-head attention mechanism is used to fuse the relationship feature representation. For example, different attention heads are used to focus on different feature dimensions, and then the outputs of multiple attention heads are concatenated or weighted averaged. The relationship prediction result is generated based on the fused relationship feature representation. For example, it is predicted that there is a "sub-domain" relationship between the document "Deep learning is a key technology of artificial intelligence" and the document "Machine learning is the foundation of artificial intelligence".

[0178] Finally, organize the text clustering results into a topic hierarchy and use the relationship prediction results as an entity relationship structure. For example, take "artificial intelligence" as the upper-level topic, "deep learning" and "machine learning" as the lower-level topics, and predict the relationship between "deep learning" and "artificial intelligence" as a "sub-domain" relationship. Generate a structured query report based on the topic hierarchy and entity relationship structure. For example, generate a JSON format report containing the topic hierarchy and entity relationship structure to facilitate user query and browsing.

[0179] The beneficial effects of this method are reflected in three aspects:

[0180] Improve query efficiency: By organizing text information into a structured topic hierarchy and entity relationship structure, users can quickly locate the required information and improve query efficiency.

[0181] Enhance the interpretability of query results: By displaying the topic hierarchy and entity relationship structure, the association between text information can be clearly displayed, enhancing the interpretability of query results.

[0182] Support more complex query requirements: By combining the subject hierarchy and entity relationship structure, more complex query requirements can be supported, such as joint queries based on subjects and relationships.

[0183] Figure 2 FIG. 1 is a schematic diagram of the structure of a data query system based on cross-modal similarity text mining according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0184] The first unit is used to obtain a text to be queried and a target text in a preset text library, wherein the target text includes image feature data and temporal relationship data; construct a multi-level feature extraction network, wherein the multi-level feature extraction network includes a text feature extraction module, an image feature extraction module, and a temporal feature extraction module; input the text to be queried into the text feature extraction module to obtain a text multi-dimensional feature vector, input the image feature data into the image feature extraction module to obtain an image multi-dimensional feature vector, and input the temporal relationship data into the temporal feature extraction module to obtain a temporal multi-dimensional feature vector;

[0185] The second unit is used to construct an adaptive cross-modal contrastive learning network, which includes a feature fusion subnetwork and a dynamic weight adjustment module; the text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector are input into the feature fusion subnetwork, and different modal features are fused and mapped through the dynamic weight adjustment module to obtain adaptive fusion features; and the contrastive learning loss function is used to optimize based on the adaptive fusion features to construct a cross-modal similarity matrix;

[0186] The third unit is used to construct a multi-level screening model, which includes a coarse-grained filtering layer and a fine-grained evaluation layer; using the multi-level screening model, the target text is hierarchically screened based on the cross-modal similarity matrix to obtain a target candidate text set; using a graph neural network to build a text relationship reasoning module, and establishing a text relationship graph based on the target candidate text set; using a multi-task learning framework to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report.

[0187] According to a third aspect of the embodiments of the present invention,

[0188] An electronic device is provided, comprising:

[0189] processor;

[0190] a memory for storing processor-executable instructions;

[0191] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0192] A fourth aspect of the embodiments of the present invention is:

[0193] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0194] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data query method based on cross-modal similarity text mining, characterized in that: include: Acquire a text to be queried and a target text in a preset text library, wherein the target text includes image feature data and temporal relationship data; construct a multi-level feature extraction network, wherein the multi-level feature extraction network includes a text feature extraction module, an image feature extraction module, and a temporal feature extraction module; Input the text to be queried into the text feature extraction module to obtain a text multidimensional feature vector, input the image feature data into the image feature extraction module to obtain an image multidimensional feature vector, and input the time series relationship data into the time series feature extraction module to obtain a time series multidimensional feature vector; Constructing an adaptive cross-modal contrastive learning network, the adaptive cross-modal contrastive learning network includes a feature fusion subnetwork and a dynamic weight adjustment module; inputting the text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector into the feature fusion subnetwork, fusing and mapping different modal features through the dynamic weight adjustment module to obtain adaptive fusion features; optimizing based on the adaptive fusion features using a contrastive learning loss function to construct a cross-modal similarity matrix; A multi-level screening model is constructed, wherein the multi-level screening model includes a coarse-grained filtering layer and a fine-grained evaluation layer; the target text is hierarchically screened based on the cross-modal similarity matrix using the multi-level screening model to obtain a target candidate text set; a text relationship reasoning module is constructed using a graph neural network, and a text relationship graph is established based on the target candidate text set; a multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report.

2. The method according to claim 1, characterized in that Construct a multi-level feature extraction network, including: Constructing a hierarchical attention network of a text feature extraction module, converting the query text into an initial word vector through a word vector embedding layer; constructing a self-attention matrix based on the initial word vector, wherein the self-attention matrix obtains a word-level attention score through a scaled dot product operation of a query vector, a key vector, and a value vector; normalizing the word-level attention score and performing a weighted combination with the value vector to obtain a word-level context representation; dividing the text data into a plurality of sentence units, performing a pooling operation on the word-level features in each sentence unit to obtain a sentence representation; constructing an inter-sentence attention network to calculate the semantic dependency between sentences, and obtaining a multi-dimensional feature vector of the text; Constructing a spatial attention network of an image feature extraction module, extracting a multi-scale feature map of the image feature data through a multi-layer convolutional network; constructing a channel attention branch and a spatial attention branch for the multi-scale feature map; in the channel attention branch, obtaining a channel descriptor through global average pooling and maximum pooling operations, and inputting the channel descriptor into a multi-layer perceptron to generate a channel weight; in the spatial attention branch, generating a two-dimensional attention map through a convolution operation; performing feature recalibration on the channel weight and the two-dimensional attention map to obtain a multi-dimensional feature vector of the image; A bidirectional long short-term memory network of a temporal feature extraction module is constructed to map the timestamp information of the temporal relationship data into a time embedding vector; a forward long short-term memory network layer and a backward long short-term memory network layer are constructed, and both the forward long short-term memory network layer and the backward long short-term memory network layer include an input gating unit, a forgetting gating unit and an output gating unit; the input gating unit is used to control the current moment information input ratio, the forgetting gating unit is used to adjust the degree of historical information retention, and the output gating unit is used to determine the state output information amount; a temporal attention layer is constructed on top of the long short-term memory network layer, the correlation strength of different time steps is calculated, and the historical states are weightedly combined according to the attention weights to obtain a multi-dimensional feature vector of the image.

3. The method according to claim 1, characterized in that Construct an adaptive cross-modal contrastive learning network, including: Constructing a feature fusion subnetwork, dividing the feature fusion subnetwork into a text feature transformation branch, an image feature transformation branch, and a temporal feature transformation branch; setting a projection layer in each feature transformation branch, wherein the projection layer includes a fully connected layer for feature dimension transformation, a batch normalization layer for feature distribution standardization, and an activation function for nonlinear mapping; setting a cross-modal attention layer after the projection layer, wherein the cross-modal attention layer includes a query mapping unit, a key-value mapping unit, and an attention calculation unit; The input text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector are respectively input into the projection layers of the text feature transformation branch, the image feature transformation branch and the time series feature transformation branch; the projection layer maps the feature vectors of different dimensions to the feature space of the same dimension through the fully connected layer, the batch normalization layer and the activation function to obtain a unified feature vector; the cross-modal attention layer uses the unified feature vector of each modality as a query vector and the unified feature vectors of other modalities as a key-value pair to calculate the inter-modal attention weights, and obtains the attention-enhanced modal features; A dynamic weight adjustment module is constructed, and the dynamic weight adjustment module includes a feature quality assessment network and a weight generation network; the feature quality assessment network receives the statistics and distribution information of the attention-enhanced modal features, and outputs a feature reliability score; the weight generation network generates dynamic fusion weights based on the feature reliability score and the attention-enhanced modal features through a multi-layer perceptron, and normalizes the dynamic fusion weights; the dynamic fusion weights are weightedly combined with the attention-enhanced modal features to obtain an adaptive fusion feature.

4. The method according to claim 1, characterized in that The contrastive learning loss function is used to optimize based on the adaptive fusion features to construct a cross-modal similarity matrix, including: Performing data enhancement processing on the adaptive fusion features to generate positive sample pairs, and randomly sampling from the adaptive fusion features of the same batch to obtain negative sample pairs; constructing a single modality contrast loss based on the positive sample pairs and the negative sample pairs; constructing a modality alignment loss and a modality contrast loss, wherein the modality alignment loss realizes feature alignment by minimizing the feature distance of the same adaptive fusion feature in different modalities, and the modality contrast loss realizes feature discrimination by increasing the feature distance between different adaptive fusion features, thereby obtaining an optimized discrimination feature; Based on the spatial local density of the optimized discriminant feature, a dynamic neighborhood is constructed for each query sample; in the dynamic neighborhood, the correlation weight between the sample and other samples in the neighborhood is calculated by the attention mechanism to obtain the neighborhood attention weight; and the cosine similarity of the sample pair in the optimized discriminant feature space is calculated as the initial similarity; A graph attention network is constructed, and the neighborhood attention weights and the initial similarities are input into the graph attention network for feature propagation and aggregation to obtain a similarity representation that integrates local structural information; the similarity representation is iteratively updated to obtain a final similarity, and a cross-modal similarity matrix is ​​constructed based on the final similarities of all sample pairs, wherein the constructed cross-modal similarity matrix includes direct similarity relationships and indirect similarity information based on local structures.

5. The method according to claim 4, characterized in that Construct a graph attention network, including: The input samples are constructed into a graph structure, in which nodes represent sample features and edges represent the strength of association between samples; an adjacency matrix of the graph is constructed based on pre-acquired neighborhood attention weights, and each node is connected to nodes in its dynamic neighborhood; the node features are mapped to multiple semantic subspaces through a multi-head attention mechanism, and each semantic subspace is used to independently learn feature association patterns; For the node pairs in the graph structure, the node features are input into the weight matrix for linear transformation to obtain a query vector and a key vector, wherein the query vector is used to represent the feature query information of the target node, and the key vector is used to represent the feature response information of the source node; the query vector and the key vector are matrix multiplied to obtain the original attention score; the original attention score is element-wise multiplied with the pre-calculated neighborhood attention weight to obtain the local structure constraint score; the local structure constraint score is nonlinearly transformed through an activation function with a negative slope, and normalized using a function to obtain the attention coefficient between the node pairs; A weighted matrix is ​​constructed based on the attention coefficient, and matrix multiplication operation is performed on the weighted matrix and the neighborhood features of the node to obtain aggregated features; the aggregated features are spliced ​​with the original features of the node in the feature dimension, and nonlinear feature transformation is performed through a multi-layer perceptron to obtain preliminary updated features; the preliminary updated features are residually connected with the original features of the node, and the feature distribution is adjusted through a layer normalization network to obtain standardized features; the standardized features are selectively retained through an adaptive gating mechanism to obtain the effective information of the original features and the updated features, so as to obtain a similarity representation that integrates local structural information.

6. The method according to claim 1, characterized in that The target text is hierarchically screened by using the multi-level screening model based on the cross-modal similarity matrix to obtain a target candidate text set, including: Performing bidirectional normalization processing on the cross-modal similarity matrix, wherein the bidirectional normalization processing includes normalizing the row vectors and applying a normalization function to the column vectors to obtain a preliminary normalized similarity matrix; calculating K average similarities between each text and its neighbors as local similarity features, and performing weighted adjustment on the preliminary normalized similarity matrix based on the local similarity features to obtain a similarity matrix that strengthens the local structure; The similarity matrix of the enhanced local structure is mapped to a low-dimensional space and subjected to product quantization discretization processing to construct a multi-layer index tree structure; an adaptive search radius is set based on the multi-layer index tree structure, an approximate nearest neighbor search is performed within the adaptive search radius, and search path information is recorded to obtain a preliminary screening candidate text; features are extracted from the preliminary screening candidate text at multiple semantic levels and feature similarity scores are calculated, and the feature similarity scores are merged with the search path information to obtain a multi-dimensional evaluation index; a weighted ranking model is constructed based on the multi-dimensional evaluation index, and the preliminary screening candidate text is re-ranked to obtain a re-ranked candidate text; The similarity distribution characteristics of the reordered candidate texts are counted, and the screening threshold is adaptively adjusted based on the similarity distribution characteristics; the similarity between the reordered candidate texts is calculated, and the diversity constraints are performed on the reordered candidate texts based on the maximum marginal correlation algorithm to obtain an intermediate candidate text set; the confidence score of each candidate text in the intermediate candidate text set is calculated based on the feature consistency of multiple modalities; the candidate texts with confidence scores higher than a preset threshold are preferentially retained to obtain a target candidate text set.

7. The method according to claim 1, characterized in that A multi-task learning framework is used to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report, including: Perform word-level feature extraction, sentence-level feature extraction and document-level feature extraction on the text node to obtain word-level features, sentence-level features and document-level features; perform weighted fusion on the word-level features, sentence-level features and document-level features to obtain text node features; based on the text node features, perform feature extraction on the relationship edge using bidirectional feature propagation to obtain initial relationship edge features; calculate information aggregation weights based on the initial relationship edge features, perform weighted aggregation on the initial relationship edge features, and obtain optimized relationship edge features; The text node features and the optimized relationship edge features are input into a multi-task learning framework, which includes a text clustering subtask and a relationship prediction subtask; for the text clustering subtask, a node similarity matrix is ​​calculated based on the text node features; for the relationship prediction subtask, an edge feature vector is calculated based on the optimized relationship edge features; based on the node similarity matrix, a probability distribution from a sample to a cluster center is calculated, and a distance metric function is constructed in combination with the graph structure information of the text relationship graph; the text nodes are clustered according to the distance metric function and the probability distribution to obtain a text clustering result; Based on the edge feature vector, a relation feature representation is constructed from the semantic dimension, the structural dimension and the temporal dimension, and the relation feature representation is fused using a multi-head attention mechanism; a relation prediction result is generated according to the fused relation feature representation; the text clustering result is organized into a topic hierarchy, and the relation prediction result is used as an entity relationship structure; and a structured query report is generated based on the topic hierarchy and the entity relationship structure.

8. A cross-modal similarity-based text mining data query system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain the target text in the query text and the preset text library, wherein the target text includes image feature data and time series relationship data; construct a multi-level feature extraction network, wherein the multi-level feature extraction network includes a text feature extraction module, an image feature extraction module and a time series feature extraction module; Input the text to be queried into the text feature extraction module to obtain a text multidimensional feature vector, input the image feature data into the image feature extraction module to obtain an image multidimensional feature vector, and input the time series relationship data into the time series feature extraction module to obtain a time series multidimensional feature vector; The second unit is used to construct an adaptive cross-modal contrastive learning network, which includes a feature fusion subnetwork and a dynamic weight adjustment module; the text multidimensional feature vector, the image multidimensional feature vector and the time series multidimensional feature vector are input into the feature fusion subnetwork, and different modal features are fused and mapped through the dynamic weight adjustment module to obtain adaptive fusion features; and the contrastive learning loss function is used to optimize based on the adaptive fusion features to construct a cross-modal similarity matrix; The third unit is used to construct a multi-level screening model, which includes a coarse-grained filtering layer and a fine-grained evaluation layer; using the multi-level screening model, the target text is hierarchically screened based on the cross-modal similarity matrix to obtain a target candidate text set; using a graph neural network to build a text relationship reasoning module, and establishing a text relationship graph based on the target candidate text set; using a multi-task learning framework to perform text clustering and relationship prediction on the text relationship graph to generate a structured query report.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images