A library cross-modal retrieval method based on semantic understanding

Abstract

The application discloses a library cross-modal retrieval method based on semantic understanding and belongs to the technical field of library information retrieval and cross-modal artificial intelligence. The method comprises the following steps: acquiring user query text and a library multi-modal resource library containing book cover images, abstract texts, author audio clips and directory structure images; performing multi-dimensional semantic graph construction processing on the query text and each resource to generate a query semantic graph and a resource semantic graph; calling a pre-trained causal inference network model to perform cross-modal causal correlation reasoning on the query semantic graph and the resource semantic graph to generate a resource correlation degree matrix; and performing root factor graph retrieval on the multi-modal resource library according to the resource correlation degree matrix to output a target retrieval result set. The application uniformly represents heterogeneous modalities through multi-dimensional semantic graphs and effectively improves the precision and interpretability of cross-modal retrieval by mining deep semantic correlations through causal inference.

Description

Technical Field

[0001] This invention relates to the fields of library information retrieval and cross-modal artificial intelligence, specifically a cross-modal library retrieval method based on semantic understanding. Background Technology

[0002] Library collections are increasingly exhibiting multimodal characteristics, encompassing various forms such as book cover images, abstract texts, author audio clips, and catalog structure images. Users often need to retrieve relevant information across modalities during searches. Existing cross-modal retrieval methods in libraries primarily rely on mapping different modalities to a common feature space and calculating feature similarity to complete the retrieval. These approaches generally suffer from insufficient characterization of deep semantic relationships, failing to reveal the implicit causal logic between query intent and resource content, resulting in retrieval results remaining at a superficial relevance matching level. Another drawback is that multimodal data are isolated during feature extraction; images, text, audio, and structural images are processed independently, lacking a unified representation structure capable of carrying fine-grained semantic associations across modalities. This makes it difficult to fully integrate and utilize complementary information between different modalities. As users' demands for retrieval depth and accuracy continue to increase, similarity-centric retrieval paradigms show significant limitations in handling complex queries, implicit intents, and multi-hop semantic reasoning. Therefore, how to construct a knowledge representation that can integrate multiple modalities and retain rich semantic associations, and how to mine the deep matching relationship between queries and resources from the perspective of causal association, have become the problems that urgently need to be solved in this technical direction. Summary of the Invention

[0003] This invention aims to provide a cross-modal library retrieval method based on semantic understanding, which solves the problems of multimodal semantic fragmentation and lack of causal reasoning ability in existing methods, realizes unified semantic representation and deep association discovery of heterogeneous library resources, and improves the accuracy and interpretability of cross-modal retrieval.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a cross-modal library retrieval method based on semantic understanding, the method comprising:

[0005] The system retrieves user query text and a multimodal resource database from the library, which includes book cover images, abstract text, author audio clips, and catalog structure images.

[0006] Perform multidimensional semantic graph construction processing on the user query text and each resource in the multimodal resource library to generate a query semantic graph and a resource semantic graph corresponding to each resource;

[0007] A pre-trained causal inference network model is invoked to perform cross-modal causal association inference processing on the query semantic graph and the resource semantic graph, generating a resource association degree matrix;

[0008] Based on the resource correlation matrix, root factor graph retrieval processing is performed on the multimodal resource database to generate a target retrieval result set.

[0009] As a preferred technical solution of the present invention, the multidimensional semantic graph construction process includes: performing intent entity extraction processing on the user query text to obtain a query intent entity set; performing visual semantic segmentation processing on the cover image of each resource in the multimodal resource library to obtain an image region semantic label set; performing syntactic dependency parsing processing on the summary text of each resource to obtain a text semantic role sequence; performing voiceprint sentiment annotation processing on the author audio segments of each resource to obtain a sentiment prosodic feature vector; performing hierarchical structure recognition processing on the directory structure image of each resource to obtain a chapter semantic tree; constructing the query semantic graph by using the query intent entity set as nodes and the semantic associations between entities as edges; and constructing the resource semantic graph by using the image region semantic label set, the text semantic role sequence, the sentiment prosodic feature vector, and the chapter semantic tree as multi-type nodes and the semantic similarity between different types of nodes as edges. By unifying the expression of heterogeneous resources such as images, text, audio, and structural images in the library into a multidimensional semantic graph, structured semantic extraction and alignment of content from different modalities are achieved, providing a unified graph structure foundation for cross-modal deep association reasoning.

[0010] Furthermore, the step of calling a pre-trained causal inference network model to perform cross-modal causal association inference processing on the query semantic graph and the resource semantic graph to generate a resource association matrix includes: performing graph embedding encoding on the query semantic graph to obtain a query semantic embedding vector; performing graph embedding encoding on the resource semantic graph to obtain a resource semantic embedding vector; inputting the query semantic embedding vector and the resource semantic embedding vector into the cross-modal contrastive learning layer of the causal inference network model for interactive matching processing to generate an initial matching score vector; and performing multi-hop causal path mining processing on the initial matching score vector through the causal path discovery layer of the causal inference network model to generate the resource association matrix, where each element in the resource association matrix represents a causal association strength score between the user query text and a resource in the multimodal resource library. This scheme replaces traditional relevance matching with causal inference, enabling the discovery of multi-hop causal chains between query intent and resource content, significantly improving the interpretability and reliability of cross-modal retrieval results.

[0011] In a preferred embodiment of the present invention, the graph embedding encoding process includes: performing subgraph sampling on the query semantic graph to obtain multiple query subgraphs; calling a pre-trained graph attention network model to perform heterogeneous node aggregation on each query subgraph to generate a query subgraph feature vector; and performing hierarchical pooling on each query subgraph feature vector to obtain the query semantic embedding vector. The hierarchical pooling process includes organizing each query subgraph feature vector according to a preset hierarchical structure to generate a multi-level feature pyramid, performing step-by-step dimensionality reduction and fusion on the multi-level feature pyramid, and outputting the query semantic embedding vector. The same graph embedding encoding method is used to obtain resource semantic embedding vectors for the resource semantic graph. By combining subgraph sampling with hierarchical pyramid fusion, the local fine-grained features and global structural features of the semantic graph are effectively captured, enhancing the expressive power and discriminative power of the embedding vector.

[0012] As a preferred embodiment of the present invention, the interaction matching process includes: calculating the cosine similarity between the query semantic embedding vector and the resource semantic embedding vector to generate basic similarity features; performing a vector outer product operation on the query semantic embedding vector and the resource semantic embedding vector to generate an interaction feature matrix; performing attention-based joint embedding processing on the basic similarity features and the interaction feature matrix to generate a joint embedding feature vector; and performing matching degree scoring processing on the joint embedding feature vector through the fully connected classification layer of the causal inference network model to generate the initial matching score vector. This scheme integrates first-order linear similarity and higher-order nonlinear interaction information, and applies attention weighting to different interaction features, thereby improving the accuracy and robustness of cross-modal fine-grained matching.

[0013] Preferably, the step of performing root factor graph retrieval processing on the multimodal resource library based on the resource correlation matrix to generate a target retrieval result set includes: performing threshold filtering on the resource correlation matrix to select a list of candidate resource identifiers with correlation scores higher than a preset threshold; extracting the key causal node sequence in the corresponding resource semantic graph for each candidate resource identifier in the candidate resource identifier list; calling a pre-trained path reasoning model to perform semantic coherence verification processing on the key causal node sequence to generate a verified node sequence; and performing subgraph matching processing between the verified node sequence and the query semantic graph to generate the target retrieval result set, which contains library resource information that is most closely related to the semantic causal relationship of the user's query text. Through the constraint filtering and path reasoning verification of the root factor graph structure, semantically fragmented or incoherent resources can be effectively removed, ensuring a high degree of consistency in the semantic causal level of the finally returned resources.

[0014] Furthermore, the semantic coherence verification process includes: inputting the key causal node sequence into the sequence encoder of the path reasoning model for node position encoding to generate an encoded node sequence; calling the conditional random field layer of the path reasoning model to calculate the label transition probability of the encoded node sequence to generate a node coherence probability distribution; and removing nodes with probabilities below a preset threshold from the key causal node sequence based on the node coherence probability distribution to obtain the verified node sequence. Using conditional random fields to model the global coherence of the node sequence can accurately identify and filter out noisy nodes in the causal chain, improving the accuracy of subsequent subgraph matching.

[0015] Furthermore, the subgraph matching process includes: performing subgraph reconstruction processing on the verified node sequence to generate candidate resource subgraphs; calculating the structural similarity score and semantic similarity score between the candidate resource subgraphs and the query semantic graph; performing weighted summation on the structural similarity score and the semantic similarity score to obtain a comprehensive matching score; and sorting the resources in the candidate resource identifier list based on the comprehensive matching score, selecting the top N resources as the target retrieval result set. Through the joint evaluation of structural similarity and semantic similarity, the retrieval result ranking is made more aligned with the user's actual information needs, improving the hit rate of multimodal resources.

[0016] As another preferred embodiment of the present invention, the method for constructing the pre-trained causal inference network model includes: acquiring a cross-modal sample dataset, each set of sample data including sample query text, sample multimodal resources, and corresponding sample causal association labels; performing the multidimensional semantic graph construction process on each resource in the sample query text and the sample multimodal resources to obtain a sample query semantic graph and a sample resource semantic graph; performing the graph embedding encoding process on the sample query semantic graph and the sample resource semantic graph to obtain a sample query embedding vector and a sample resource embedding vector; inputting the sample query embedding vector and the sample resource embedding vector into an initial causal inference network, the initial causal inference network including a cross-modal contrastive learning layer and a causal path discovery layer, outputting an initial sample matching score vector through the cross-modal contrastive learning layer, and outputting a sample resource association degree prediction matrix through the causal path discovery layer; calculating a joint loss function value based on the sample resource association degree prediction matrix and the sample causal association labels, and performing backpropagation and parameter iterative updates on the initial causal inference network until convergence to obtain the pre-trained causal inference network model. Supervised causal labeling guides the model to learn an accurate mapping from semantic graphs to the strength of causal associations, ensuring the authenticity of the inference results.

[0017] Further preferably, the method further includes: parsing the resource semantic graph corresponding to each search result in the target search result set, and extracting the modality type node distribution in the resource semantic graph; generating a cross-modal search feedback signal based on the modality type node distribution, wherein the cross-modal search feedback signal is used to indicate the contribution ratio of each modality resource in the search results to the user query; inputting the cross-modal search feedback signal and the resource correlation matrix into a policy optimization network model for reinforcement learning policy update processing, and generating updated causal inference network model parameters, specifically including: encoding the cross-modal search feedback signal into a reward value vector; performing feature concatenation processing on the reward value vector and the resource correlation matrix to generate a state feature vector; calling the policy gradient calculation layer of the policy optimization network model to perform gradient descent calculation processing on the state feature vector to generate model parameter update amount; and using the model parameter update amount to perform batch normalization adjustment processing on the weight parameters of the causal inference network model to generate the updated causal inference network model parameters. This feedback update mechanism enables the causal inference network to continuously strengthen and optimize based on the actual multimodal contribution distribution, thereby dynamically adapting to updates to the library resource database and changes in user search behavior, and maintaining a high level of cross-modal search performance over a long period of time.

[0018] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0019] By transforming query text and multimodal resources into multidimensional semantic graphs, this method extracts intent entity sets from the query text as nodes and generates edges based on semantic relationships between entities. Visual semantic segmentation is performed on book cover images to generate image region semantic labels; syntactic dependency parsing is performed on summary text to generate text semantic role sequences; voiceprint sentiment annotation is performed on author audio clips to obtain sentiment prosodic feature vectors; and hierarchical structure recognition is performed on catalog structure images to construct chapter semantic trees. These semantic elements from different modalities are unified as multi-type nodes, and cross-modal edges are connected according to their semantic similarity, ultimately constructing query semantic graphs and resource semantic graphs. This graph structure can organize scattered modal semantics into a unified representation with entities, relationships, and attributes as its skeleton. It preserves the fine-grained correspondence between image region semantic labels and text semantic roles, and the contextual association between sentiment prosodic features and chapter semantic trees. It eliminates the isolation of modal information processing in traditional solutions, allowing subsequent reasoning to be performed directly in the semantic graph space containing multimodal knowledge, significantly enhancing the completeness and structure of semantic understanding of library multimodal resources.

[0020] After obtaining the query semantic graph and resource semantic graph, they are compressed into vector representations through graph embedding encoding. These vector representations are then fed into the cross-modal contrastive learning layer of the causal inference network model to calculate interaction features and generate an initial matching score. The causal path discovery layer then mines multiple causal paths from the initial matching score to obtain a resource association matrix. Each element in this matrix reflects the strength of the causal association between the query and the corresponding resource. Subsequently, threshold filtering is performed based on the resource association matrix to obtain a candidate resource identifier list. Key causal node sequences corresponding to the candidate resources are extracted, and the path inference model is invoked to verify the semantic coherence of the sequences. Nodes with low coherence are eliminated. Subgraph matching is then performed between the verified node sequences and the query semantic graph. The target retrieval result set is output by ranking the results based on both structural and semantic similarity scores. This process goes beyond shallow matching based on relevance. Through explicit causal path discovery and multi-hop inference, it filters content from massive resources that has a causal semantic chain with the query, reducing the proportion of invalid matches and making the retrieval results more interpretable, thus adapting to the deep semantic retrieval needs in complex query scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart of a semantic understanding-based cross-modal library retrieval method;

[0023] Figure 2 This is a flowchart of the root factor graph retrieval process for a multimodal resource library;

[0024] Figure 3 This is a flowchart of the pre-trained causal inference network model construction process;

[0025] Figure 4 This is a schematic diagram of the training and early stopping of a causal inference network model. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] See Figure 1This invention provides a cross-modal library retrieval method based on semantic understanding. The method acquires a user query text and a multimodal library resource database, including book cover images, abstract text, author audio clips, and a table of contents image. Multidimensional semantic graph construction is performed on the user query text and each resource in the multimodal resource database, generating a query semantic graph and a corresponding resource semantic graph. A pre-trained causal inference network model is invoked to perform cross-modal causal association inference on the query semantic graph and the resource semantic graph, generating a resource association degree matrix. Based on the resource association degree matrix, root factor graph retrieval is performed on the multimodal resource database to generate a target retrieval result set.

[0028] Example 1:

[0029] In practical implementation, the multidimensional semantic graph construction process includes processing flows for user query text and processing flows for each resource in the multimodal resource library. In the user query text processing flow, intent entity extraction is performed on the user query text to obtain a query intent entity set. Intent entity extraction is implemented through a pre-trained intent recognition model and a named entity recognition model. The pre-trained intent recognition model uses a bidirectional long short-term memory network combined with an attention mechanism, taking the word embedding sequence of the user query text as input and outputting intent category labels. The pre-trained named entity recognition model uses a transformer-based sequence labeling architecture, outputting an entity category label or non-entity label for each word in the user query text. The intent category labels and the words corresponding to all entity category labels, and their combinations, are used as the query intent entity set.

[0030] The processing of each resource in the multimodal resource library comprises four parallel branches. The first branch performs visual semantic segmentation on the book cover image, obtaining a set of semantic labels for the image regions. Visual semantic segmentation is performed using a pre-trained semantic segmentation model, which employs an encoder-decoder structure based on a convolutional neural network. The encoder extracts multi-scale feature maps from the book cover image, and the decoder upsamples these feature maps and outputs the semantic category of each pixel. Pixel regions with the same semantic category and spatial connectivity are merged into a single image region, and a corresponding semantic label is assigned to each image region. All semantic labels form the set of semantic labels for the image regions.

[0031] The second branch performs syntactic dependency parsing on the abstract text to obtain a sequence of semantic roles. This parsing is performed using a pre-trained dependency parser, which employs a transformer-based graph parsing architecture. The parser outputs the headword index and dependency relation type for each term in the abstract text. Based on the dependency relation types, the core predicates and their associated agent, patient, temporal, and locative roles are identified in the abstract text. These core predicates and their associated semantic roles are then organized into a sequence of semantic roles according to the text's order.

[0032] The third branch performs voiceprint sentiment annotation on the author's audio segments to obtain a sentiment prosodic feature vector. Voiceprint sentiment annotation is performed using a pre-trained sentiment recognition model, which includes a voiceprint encoding network and a sentiment classification network. The voiceprint encoding network inputs the Mel-frequency cepstral coefficient feature sequence of the author's audio segments into an encoder based on convolutional and gated recurrent units, generating a fixed-dimensional voiceprint embedding vector. The sentiment classification network takes the voiceprint embedding vector as input, outputs a sentiment category probability distribution through a fully connected layer, selects the sentiment category with the highest probability as the sentiment label, and concatenates the voiceprint embedding vector with the unique encoding of the sentiment category label to obtain the sentiment prosodic feature vector.

[0033] The fourth branch performs hierarchical structure recognition processing on the directory structure image to obtain the chapter semantic tree. Hierarchical structure recognition is performed using a pre-trained document structure analysis model. This model employs a network architecture based on object detection to detect text line regions in the directory structure image and identify the hierarchical label and text content for each text line region. Based on the parent-child relationship of the hierarchical labels, the text content is organized into a multi-branch tree structure. Each node in the tree stores a chapter title text and its corresponding hierarchical depth, ultimately outputting the chapter semantic tree.

[0034] After obtaining the query intent entity set, the image region semantic label set, text semantic role sequence, sentiment prosody feature vector, and chapter semantic tree for each resource, a semantic graph is constructed. The query semantic graph is constructed as follows: each entity in the query intent entity set is treated as a node in the query semantic graph. The semantic association strength between any two entities is calculated, represented by the cosine similarity between entity vectors obtained from a pre-trained word vector model. When the cosine similarity between two entities exceeds a preset similarity threshold, an edge is created between these two entity nodes in the query semantic graph, with the edge weight being the cosine similarity value. The preset similarity threshold is 0.45, selected based on optimal connectivity of the semantic graph achieved through adjustments on the validation set. All nodes and edges in the query semantic graph constitute the query semantic graph.

[0035] The resource semantic graph is constructed as follows: each semantic label in the image region semantic label set is treated as an image node; each semantic role in the text semantic role sequence is treated as a text node; the sentiment prosody feature vector is treated as a sentiment node; and each chapter title in the chapter semantic tree is treated as a chapter node. Image nodes, text nodes, sentiment nodes, and chapter nodes all serve as nodes in the resource semantic graph, with the node type attribute labeled as the corresponding modality type. For any two nodes of different types, the semantic similarity between the nodes is calculated using a multimodal semantic alignment function.

[0036]

[0037] in, and These represent two different types of nodes in the resource semantic graph; and Representing nodes respectively and nodes The embedding vector after being encoded by the corresponding modal encoder express Transpose of; Represents nodes The set of all other nodes of different types; This is the temperature scaling factor, with a value range of [value range missing]. The value is determined on the validation set through a grid search, specifically set to 1.2. This applies to nodes of different types. When the preset multimodal connectivity threshold is exceeded, it becomes a node in the resource semantic graph. and nodes Establish an edge between them, and take the weight of the edge. The value is set to 0.35, based on the premise that the average node degree of the resource semantic graph is within a preset range. All nodes and edges together constitute the resource semantic graph.

[0038] Example 2:

[0039] In practice, a pre-trained causal inference network model is invoked to perform cross-modal causal association reasoning on the query semantic graph and the resource semantic graph. The process of generating the resource association degree matrix is ​​completed collaboratively through graph embedding encoding, cross-modal contrastive learning layer interactive matching, and causal path discovery layer multi-hop causal path mining.

[0040] The pre-trained causal inference network model includes a query graph embedding encoding module, a resource graph embedding encoding module, a cross-modal contrastive learning layer, and a causal path discovery layer. The query graph embedding encoding module receives the query semantic graph and outputs query semantic embedding vectors. The processing flow of the query graph embedding encoding module is as follows: Subgraph sampling is performed on the query semantic graph to obtain multiple query subgraphs. The subgraph sampling process adopts a subgraph extraction strategy based on incomplete random walks. Starting from each node in the query semantic graph, a random walk with a step size of 3 is performed, for a total of 8 independent walks. The nodes and edges traversed in each walk constitute a query subgraph. A pre-trained graph attention network model is then called to perform heterogeneous node aggregation processing on each query subgraph to generate a query subgraph feature vector. The pre-trained graph attention network model contains two graph attention convolutional layers. Each layer calculates the attention coefficient between each node in the query subgraph and its neighboring nodes, and then aggregates the features of the neighboring nodes into the representation of the current node based on the attention coefficient. The output dimension of the first layer is 128, and the output dimension of the second layer is 256. The average of the representation vectors of all nodes in the query subgraph after passing through the second layer is calculated to obtain the 256-dimensional query subgraph feature vector. Hierarchical pooling is then performed on each query subgraph feature vector to obtain the query semantic embedding vector. Hierarchical pooling organizes the query subgraph feature vectors according to a predefined hierarchical structure, generating a multi-level feature pyramid. This multi-level feature pyramid is constructed by progressively aggregating the feature vectors of adjacent query subgraphs. Specifically, all query subgraph feature vectors are used as the bottom layer of the pyramid, and non-overlapping average pooling is performed using a sliding window of size 2 to obtain the features of the next layer. This process is repeated until only a single vector remains at the top layer, which is the 512-dimensional query semantic embedding vector.

[0041] The resource graph embedding encoding module receives a resource semantic graph and outputs a resource semantic embedding vector. The resource graph embedding encoding module uses the same network structure as the query graph embedding encoding module. First, it performs subgraph sampling on the resource semantic graph to obtain multiple resource subgraphs. Then, it calls a pre-trained graph attention network model to perform heterogeneous node aggregation on each resource subgraph to generate a resource subgraph feature vector. Finally, it fuses the feature vectors of each resource subgraph into a 512-dimensional resource semantic embedding vector through hierarchical pooling.

[0042] The query semantic embedding vector and resource semantic embedding vector are input into the cross-modal contrastive learning layer of the causal inference network model for interactive matching processing to generate an initial matching score vector. The cross-modal contrastive learning layer includes a basic similarity calculation module, an interaction feature construction module, an attention-based joint embedding module, and a fully connected classification module. The basic similarity calculation module calculates the cosine similarity between the query semantic embedding vector and the resource semantic embedding vector, generating a scalar value as the basic similarity feature. The interaction feature construction module performs a vector outer product operation on the query semantic embedding vector and the resource semantic embedding vector, generating a 512×512 dimensional interaction feature matrix. This matrix is ​​then flattened into a 262144-dimensional vector, reduced to 256 dimensions by a fully connected layer, yielding the interaction feature vector. The attention-based joint embedding module performs attention-based joint embedding processing on the basic similarity feature and the interaction feature vector, generating a joint embedding feature vector. The attention-based joint embedding process is as follows: the basic similarity features are expanded into a 256-dimensional vector through linear projection, element-wise added to the interaction feature vector, and then passed through a multi-head self-attention layer to obtain the joint embedding feature vector. The multi-head self-attention layer has 4 heads, and the output dimension remains 256. The fully connected classification module contains two fully connected layers. The first fully connected layer maps the 256-dimensional joint embedding feature vector to 64 dimensions, and the second fully connected layer maps the 64-dimensional vector to a 1-dimensional matching score. The above calculation is performed on each resource in the multimodal resource library, and its matching score is the component in the initial matching score vector corresponding to that resource.

[0043] The causal path discovery layer of the causal inference network model performs multi-hop causal path mining on the initial matching score vector to generate a resource correlation matrix. The causal path discovery layer employs a path inference network based on conditional random fields. This network receives the encoded initial matching score vector and structural information from the query semantic graph as additional inputs. The path inference network includes a path construction module and a path scoring module. The path construction module starts from each entity node in the query semantic graph and traverses multiple hops along the edges. The probability of selecting the next node at each hop depends on the edge weight and the relevance of the current resource to the corresponding concept in the resource semantic graph, generating multiple causal paths. The path scoring module calculates a coherence score for each causal path and merges the path coherence score with the initial matching score using the following formula:

[0044]

[0045] in, Indicates the user's query text Resources in the multimodal resource library The strength of the causal relationship between them; Representing resources The corresponding initial matching score; For the set of causal paths constructed from the query semantic graph, This indicates the total number of causal paths; express A causal path in; causal path The number of jumps; It is a causal path In query semantic graph Above and Resources The semantic coherence function between the resource semantic graph structures is calculated by inputting the embedding sequence of nodes on the path into the long short-term memory network to obtain the coherence output value. and As a balance factor, The value is 0.7. The value is 0.3, and the value is determined by tuning the normalized loss cumulative gain index on the validation set during model training. This is the attenuation coefficient, with a value of 0.5, used to penalize long paths; Using the sigmoid activation function, the score is mapped to... Interval. The causal correlation strength scores corresponding to all resources constitute the resource correlation matrix.

[0046] Example 3:

[0047] In specific implementation, please refer to Figure 2 The root factor graph retrieval process is performed on the multimodal resource library based on the resource correlation matrix. The operation of generating the target retrieval result set is completed through four steps: threshold filtering, key causal node sequence extraction, semantic coherence verification, and subgraph matching.

[0048] A threshold filtering process is applied to the resource correlation matrix to select a list of candidate resource identifiers with correlation scores higher than a preset threshold. Each element in the resource correlation matrix represents the causal association strength score between the user's query text and a resource in the multimodal resource library. The preset threshold is set to 0.62, which was determined by maximizing the F1 score of the search results during the offline model evaluation phase. The value of each element in the resource correlation matrix is ​​compared one by one with the preset threshold of 0.62, and resource identifiers corresponding to resources with scores greater than 0.62 are retained. These resource identifiers form the candidate resource identifier list. Each candidate resource identifier in the candidate resource identifier list uniquely corresponds to a resource in the multimodal resource library.

[0049] For each candidate resource identifier in the candidate resource identifier list, extract the key causal node sequence from the resource semantic graph corresponding to the candidate resource identifier. The process for extracting the key causal node sequence is as follows: obtain all causal paths generated by the causal path discovery layer during the generation of the resource relevance score corresponding to the resource; select the causal path with the highest path coherence score from all causal paths; and arrange the nodes on this causal path in path order to form the key causal node sequence. Each node in the key causal node sequence belongs to a node in the resource semantic graph, and the original type attribute of the node is preserved.

[0050] A pre-trained path reasoning model is invoked to perform semantic coherence verification on the key causal node sequence, generating a sequence of nodes that pass the verification. The pre-trained path reasoning model includes a sequence encoder and a conditional random field (CRF) layer. The sequence encoder adopts a transformer-based encoding architecture, containing a multi-head self-attention layer and a feedforward network layer. The input to the sequence encoder is the key causal node sequence, where each node is represented as its embedding vector, which is obtained from the node embeddings during the resource semantic graph construction phase. The sequence encoder performs node position encoding on the key causal node sequence, generating an encoded node sequence. Specifically, for each node in the key causal node sequence, its embedding vector is added to the learnable position encoding vector. This is then passed through six stacked transformer encoding layers in the sequence encoder, with each layer outputting a 512-dimensional hidden state vector, ultimately outputting the encoded node sequence. Each vector in the encoded node sequence represents the context-aware representation of the corresponding node position. The CRF layer is then invoked to calculate the label transition probability of the encoded node sequence, generating a node coherence probability distribution. The Conditional Random Field (CRF) layer uses the encoded node sequence as the observation sequence, defining the label set as {coherent, incoherent}. The CRF layer includes state feature functions and transition feature functions. The state feature function calculates the label score for each position's hidden state vector through a linear transformation, while the transition feature function defines the transition score matrix between labels at adjacent positions. A forward-backward algorithm is used to calculate the marginal probability of each node being labeled as coherent. The coherence marginal probabilities of all nodes form the node coherence probability distribution. Based on the node coherence probability distribution, nodes with probabilities below a preset threshold are removed from the key causal node sequence, resulting in a validated node sequence. The preset coherence probability threshold is set to 0.58, determined based on the maximum Youden exponent for coherence classification on the validation set using the path inference model. The coherence marginal probability at each position in the node coherence probability distribution is compared to 0.58; nodes with probabilities greater than or equal to 0.58 are retained, and these retained nodes are arranged in their original order to form the validated node sequence.

[0051] The verified node sequence is matched with the query semantic graph using subgraph matching to generate the target retrieval result set. Subgraph matching first reconstructs the verified node sequence to generate candidate resource subgraphs. The subgraph reconstruction process involves using all nodes in the verified node sequence as the node set of the candidate resource subgraph. For any two nodes in the node set, it is checked whether an edge exists between them in the original resource semantic graph. If an edge exists, an edge is created for these two nodes in the candidate resource subgraph, with the edge weight copied from the edge weight in the original resource semantic graph. The structural similarity score and semantic similarity score between the candidate resource subgraph and the query semantic graph are calculated. The structural similarity score is obtained by normalizing the graph edit distance and graph size. The semantic similarity score is obtained by calculating the cosine similarity between the graph-level representation vector of the candidate resource subgraph and the graph-level representation vector of the query semantic graph. The graph-level representation vector is obtained by average pooling the embedding vectors of all nodes in the graph. The structural similarity score and semantic similarity score are then weighted and summed to obtain a comprehensive matching score. The formula for the weighted summation is as follows:

[0052]

[0053] in, Indicates candidate resource identifier The corresponding overall matching score; Indicates candidate resource identifier The structural similarity score between the corresponding candidate resource subgraph and the query semantic graph; Indicates candidate resource identifier The semantic similarity score between the corresponding candidate resource subgraph and the query semantic graph; and These are the weighting coefficients. The value is 0.42. The value is 0.58. The values ​​of the two weighting coefficients are determined by maximizing the average reciprocal ranking using grid search in the retrieval and ranking task. Resources in the candidate resource identifier list are sorted based on the comprehensive matching score; the higher the comprehensive matching score, the higher the ranking. The top N resources are selected as the target retrieval result set. The value of N is set to 20, which is based on the maximum number of resources that can be displayed per page in the library system. The target retrieval result set contains library resource information that has the strongest semantic causal relationship with the user's query text.

[0054] Example 4:

[0055] In specific implementation, please refer to Figure 3 The construction method of pre-trained causal inference network model includes five stages: sample data acquisition, semantic graph construction, graph embedding encoding, network forward propagation, and joint loss optimization.

[0056] A cross-modal sample dataset was obtained, with each set of sample data including sample query text, sample multimodal resources, and corresponding sample causal association labels. The sample query text was derived from reader-entered queries recorded in the library's search logs, and the text content was preserved after privacy removal. Sample multimodal resources were sampled from the library's multimodal resource repository, and each sample multimodal resource included a sample book cover image, sample abstract text, sample author audio clips, and a sample catalog structure image. The corresponding sample causal association labels were assigned by human annotation experts based on the existence of a semantic causal relationship between the sample query text and the sample multimodal resource; a 1 was assigned if a semantic causal relationship existed, and a 0 was assigned if no semantic causal relationship existed. The cross-modal sample dataset contains a total of 50,000 sets of sample data, with 40,000 sets used as the training set, 5,000 sets as the validation set, and 5,000 sets as the test set.

[0057] For each resource in the sample query text and sample multimodal resources, a multidimensional semantic graph construction process is performed to obtain the sample query semantic graph and the sample resource semantic graph. The multidimensional semantic graph construction process for the sample query text is as follows: the sample query text is subjected to intent entity extraction processing to obtain the sample query intent entity set; the sample query intent entity set is used as nodes, and the semantic relationships between entities are used as edges to construct the sample query semantic graph. The process of constructing the multidimensional semantic graph of the sample modal resources is as follows: Visual semantic segmentation is performed on the sample book cover images in the sample multimodal resources to obtain a set of semantic labels for the sample image regions; syntactic dependency parsing is performed on the sample summary text in the sample multimodal resources to obtain a sequence of semantic roles in the sample text; voiceprint sentiment annotation is performed on the sample author audio clips in the sample multimodal resources to obtain a sample sentiment prosodic feature vector; hierarchical structure recognition is performed on the sample directory structure images in the sample multimodal resources to obtain a sample chapter semantic tree; the sample image region semantic label set, the sample text semantic role sequence, the sample sentiment prosodic feature vector, and the sample chapter semantic tree are used as multi-type nodes, and the semantic similarity between different types of nodes is used as edges to construct the sample resource semantic graph.

[0058] Graph embedding encoding is performed on the sample query semantic graph and the sample resource semantic graph respectively to obtain the sample query embedding vector and the sample resource embedding vector. Graph embedding encoding includes three sub-steps: subgraph sampling, graph attention aggregation, and hierarchical pooling. In the subgraph sampling step, a subgraph extraction strategy based on incomplete random walks is executed on the input semantic graph. Starting from each node of the semantic graph, a random walk with a stride of 3 is performed, for a total of 8 independent walks. The nodes and edges traversed by each walk constitute a subgraph. In the graph attention aggregation step, a pre-trained graph attention network model is called to perform heterogeneous node aggregation processing on each subgraph. The pre-trained graph attention network model contains two graph attention convolutional layers. The first graph attention convolutional layer maps the node features to 128 dimensions, and the second graph attention convolutional layer maps the node features to 256 dimensions. The subgraph feature vector is obtained by averaging the feature vectors of all nodes in the subgraph after passing through the second graph attention convolutional layer. In the hierarchical pooling step, the feature vectors of each subgraph are organized according to a preset hierarchical structure. All subgraph feature vectors are used as the bottom layer of a multi-level feature pyramid. Non-overlapping average pooling is performed using a sliding window of size 2 to obtain the features of the next layer. This process is repeated layer by layer until only a single vector remains at the top, resulting in a 512-dimensional semantic embedding vector. The sample query semantic graph is processed by graph embedding encoding to output the sample query embedding vector, and the sample resource semantic graph is processed by graph embedding encoding to output the sample resource embedding vector.

[0059] The sample query embedding vector and sample resource embedding vector are input into the initial causal inference network, which includes a cross-modal contrastive learning layer and a causal path discovery layer. The cross-modal contrastive learning layer receives the sample query embedding vector and sample resource embedding vector and outputs an initial sample matching score vector. The cross-modal contrastive learning layer performs the following operations: It calculates the cosine similarity between the sample query embedding vector and the sample resource embedding vector to generate basic similarity features; it performs a vector outer product operation on the sample query embedding vector and the sample resource embedding vector to generate an interaction feature matrix; after flattening the interaction feature matrix, it is reduced to 256 dimensions by a fully connected layer to obtain the interaction feature vector; the basic similarity features are extended to a 256-dimensional vector through linear projection, and element-wise added to the interaction feature vector; this is then passed through a multi-head self-attention layer to generate a joint embedding feature vector. The multi-head self-attention layer has 4 heads and an output dimension of 256 dimensions; a fully connected classification layer performs matching degree scoring on the joint embedding feature vector. The fully connected classification layer contains two fully connected layers: the first fully connected layer maps the 256-dimensional vector to 64 dimensions, and the second fully connected layer maps the 64-dimensional vector to a 1-dimensional matching degree score. This process is repeated for all sample resources in a set of samples to obtain an initial sample matching score vector. The causal path discovery layer receives the initial sample matching score vector and the structural information of the sample query semantic graph, and outputs a sample resource association prediction matrix through a path construction module and a path scoring module. The path construction module starts from each entity node in the sample query semantic graph and walks along multiple causal paths. The path scoring module calculates the coherence score for each causal path and merges the path coherence score with the initial sample matching score to obtain the causal association strength prediction score for each sample resource. The causal association strength prediction scores of all sample resources constitute the sample resource association degree prediction matrix.

[0060] The joint loss function is calculated based on the sample resource correlation prediction matrix and sample causal correlation labels. Backpropagation and parameter iterative updates are then performed on the initial causal inference network until convergence, resulting in a pre-trained causal inference network model. The joint loss function consists of a matching loss term and a causal path consistency loss term, calculated as follows:

[0061]

[0062] in, Indicates the value of the joint loss function; This represents the total number of sample resources contained in each sample group; Indicates the first Each sample resource has a causal association label, which takes the value of 0 or 1. Indicates the first The causal association strength prediction score corresponding to each sample resource in the sample resource association prediction matrix, with a value range of [value range missing]. ; This indicates the number of entity nodes in the sample query semantic graph; This indicates that the causal path discovery layer generates a result related to the first... The path aggregation hidden state vector corresponding to each entity node has a dimension of 256. This indicates that the semantic graph of the sample resources, after encoding, is related to the first... A semantic hidden state vector aligned with each entity node, with a dimension of 256; Represents the square of the L2 norm; , is the adjustment coefficient for the causal path consistency loss term, with a value of 0.15. This value is determined by balancing the magnitudes of the matching loss term and the causal path consistency loss term on the validation set and by manual fine-tuning.

[0063] During the training phase, the Adam optimizer is used for backpropagation and iterative parameter updates. The initial learning rate is set to 0.0003, and the batch size is 32. An early stopping mechanism is triggered when the loss function value on the validation set does not decrease for eight consecutive training epochs. After training converges, the model weights are saved to obtain the pre-trained causal inference network model.

[0064] See Figure 4 In the graph, the horizontal axis represents the training period, ranging from 0 to 100, and the vertical axis represents the joint loss function value, ranging from 0 to 1. The solid green line represents the joint loss function value of the training set, the dashed orange line represents the joint loss function value of the validation set, and the dashed red line represents the early stopping trigger period.

[0065] As the training epochs increase, the joint loss function value of the training set generally shows a downward trend, gradually decreasing from an initial high value close to 1 to approximately 0.15, with some fluctuations but an overall decreasing trend. The joint loss function value of the validation set also shows a downward trend, gradually decreasing from approximately 0.85 to approximately 0.2, and the change curve is relatively smooth. The training set loss reaches a relatively stable state around the 50th training epoch, with subsequent fluctuations being small. The validation set loss does not show a significant decrease around the 55th training epoch, triggering the early stopping mechanism. The early stopping trigger epoch marked by the red dotted line is the 55th epoch.

[0066] Example 5:

[0067] In practice, the process of parsing the search results and generating feedback signals, and updating the parameters of the causal inference network model using reinforcement learning, is completed collaboratively by the feedback signal generation module and the policy optimization network model. The target search result set contains the top N resources, where N is 20. The resource semantic graph corresponding to each search result in the target search result set is parsed, and the modality type node distribution in the resource semantic graph is extracted. The modality type node distribution refers to the statistical vector composed of the number of nodes contained in each modality type in the resource semantic graph. The node types in the resource semantic graph include image nodes, text nodes, sentiment nodes, and chapter nodes. For each search result in the target search result set, the number of image nodes, text nodes, sentiment nodes, and chapter nodes in its resource semantic graph are counted, forming a four-dimensional vector as the modality type node distribution of that search result. The modality distribution of all search results is weighted and summed according to the ranking of the search results. The higher the ranking of the search result, the higher the weight. The weight coefficient adopts normalized discount weight. The weight of the k-th search result is 1 / log2(k+1), thus obtaining the overall modality distribution vector of the target search result set.

[0068] A cross-modal retrieval feedback signal is generated based on the modal type node distribution. This signal indicates the contribution ratio of each modal resource to the user's query in the retrieval results. The proportion of each modal component in the overall modal type node distribution vector is used as the contribution ratio to construct the cross-modal retrieval feedback signal. The cross-modal retrieval feedback signal is a four-dimensional vector, with each dimension corresponding to the contribution ratio of image modality, text modality, audio modality, and structural modality, respectively. The sum of the values ​​of the four dimensions is 1.

[0069] The cross-modal retrieval feedback signal and the resource correlation matrix are input into the policy optimization network model for reinforcement learning policy update processing, generating updated causal inference network model parameters. The policy optimization network model consists of a state encoding module, a reward encoding module, a feature fusion module, and a policy gradient calculation layer. The state encoding module receives the resource correlation matrix, flattens it to generate a one-dimensional correlation feature vector, the dimension of which equals the number of resources in the multimodal resource pool. The reward encoding module receives the cross-modal retrieval feedback signal and maps it from four dimensions to a 64-dimensional reward value vector through a fully connected layer. The activation function of the fully connected layer is a linear rectified function. The feature fusion module concatenates the reward value vector and the correlation feature vector to generate a state feature vector. The dimension of the state feature vector is the total number of resources in the resource pool plus 64.

[0070] The policy gradient calculation layer of the policy optimization network model is invoked to perform gradient descent calculation on the state feature vector, generating the model parameter update. The policy gradient calculation layer contains a parameter generation network, which consists of two fully connected layers. The first fully connected layer maps the state feature vector to a 512-dimensional hidden state vector, using a leaky linear rectified function with a leakage coefficient of 0.15 as the activation function. The second fully connected layer maps the hidden state vector to the total number of trainable parameters of the causal inference network model, outputting the original update vector. Scaling and pruning are applied to the original update vector to generate the model parameter update, calculated using the following formula:

[0071]

[0072] in, This represents the amount of model parameter updates, and its dimension is the same as the dimension of a one-dimensional vector formed by expanding all trainable parameters of the causal inference network model. This represents the global update step size, with a value of 0.008. This value is determined based on maximizing the expected value of the cumulative reward through grid search in the offline simulation environment. This represents an element-wise pruning function that restricts each component of the input vector to an interval. Inside, The pruning threshold is set to 0.05. This threshold is used to prevent the model from crashing due to excessively large single-step update magnitude. This represents the dimension-aligned projection matrix, which is learned by the policy optimization network model during the pre-training phase. It is used to convert the dimension of the output vector inside the policy gradient calculation layer into a dimension that matches the total number of parameters in the causal inference network. Represents the state feature vector; and These are the weight matrix and bias vector of the second fully connected layer in the policy gradient calculation layer, respectively; It is the hyperbolic tangent activation function.

[0073] The weight parameters of the causal inference network model are batch-normalized using the model parameter update values ​​to generate updated causal inference network model parameters. The batch normalization process is as follows: all trainable parameters of the causal inference network model are grouped by layer. For each layer's weight matrix, a sub-tensor of the corresponding shape is extracted from the model parameter update values. The weight matrix and the sub-tensor are then added element-wise. The resulting weight matrix undergoes batch normalization using the mean moving average and variance moving average obtained during the pre-training phase. A linear transformation is then applied using learnable scaling and translation parameters. The bias parameters of each layer and the parameters of non-normalized layers are directly added to the corresponding parts of the model parameter update values. After batch normalization, the updated causal inference network model parameters are obtained for use in subsequent retrieval requests.

[0074] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A cross-modal library retrieval method based on semantic understanding, characterized in that, The method includes: The system retrieves user query text and a multimodal resource database from the library, which includes book cover images, abstract text, author audio clips, and catalog structure images. Perform multidimensional semantic graph construction processing on the user query text and each resource in the multimodal resource library to generate a query semantic graph and a resource semantic graph corresponding to each resource; A pre-trained causal inference network model is invoked to perform cross-modal causal association inference processing on the query semantic graph and the resource semantic graph, generating a resource association degree matrix; Based on the resource correlation matrix, root factor graph retrieval processing is performed on the multimodal resource database to generate a target retrieval result set.

2. The library cross-modal retrieval method based on semantic understanding according to claim 1, characterized in that, The step of performing multidimensional semantic graph construction processing on the user query text and each resource in the multimodal resource library to generate a query semantic graph and a resource semantic graph corresponding to each resource includes: The user query text is subjected to intent entity extraction processing to obtain a query intent entity set; Visual semantic segmentation is performed on the cover image of each resource in the multimodal resource library to obtain a set of semantic labels for image regions; syntactic dependency parsing is performed on the summary text of each resource to obtain a text semantic role sequence; voiceprint sentiment annotation is performed on the author audio segments of each resource to obtain a sentiment prosodic feature vector; and hierarchical structure recognition is performed on the directory structure image of each resource to obtain a chapter semantic tree. The query semantic graph is constructed by using the set of entity intents as nodes and the semantic associations between entities as edges; the resource semantic graph is constructed by using the set of semantic labels for image regions, the sequence of semantic roles for text, the emotional prosody feature vector, and the chapter semantic tree as multi-type nodes and the semantic similarity between different types of nodes as edges.

3. The library cross-modal retrieval method based on semantic understanding according to claim 1, characterized in that, The process of calling a pre-trained causal inference network model to perform cross-modal causal association inference on the query semantic graph and the resource semantic graph, generating a resource association degree matrix, includes: The query semantic graph is subjected to graph embedding encoding to obtain a query semantic embedding vector; The resource semantic graph is subjected to graph embedding encoding to obtain a resource semantic embedding vector; The query semantic embedding vector and the resource semantic embedding vector are input into the cross-modal contrastive learning layer of the causal inference network model for interactive matching processing to generate an initial matching score vector; The initial matching score vector is subjected to multi-hop causal path mining processing through the causal path discovery layer of the causal inference network model to generate the resource correlation matrix. Each element in the resource correlation matrix represents the causal correlation strength score between the user query text and a resource in the multimodal resource library.

4. The library cross-modal retrieval method based on semantic understanding according to claim 3, characterized in that, The step of performing graph embedding encoding on the query semantic graph to obtain a query semantic embedding vector includes: The query semantic graph is subjected to subgraph sampling processing to obtain multiple query subgraphs; A pre-trained graph attention network model is invoked to perform heterogeneous node aggregation processing on each query subgraph to generate a query subgraph feature vector; The feature vectors of each query subgraph are subjected to hierarchical pooling to obtain the query semantic embedding vector. The hierarchical pooling process includes: organizing the feature vectors of each query subgraph according to a preset hierarchical structure to generate a multi-level feature pyramid, performing dimensionality reduction and fusion of the multi-level feature pyramid step by step, and outputting the query semantic embedding vector.

5. A cross-modal library retrieval method based on semantic understanding according to claim 3, characterized in that, The step of inputting the query semantic embedding vector and the resource semantic embedding vector into the cross-modal contrastive learning layer of the causal inference network model for interactive matching processing to generate an initial matching score vector includes: Calculate the cosine similarity between the query semantic embedding vector and the resource semantic embedding vector to generate basic similarity features; Perform a vector outer product operation on the query semantic embedding vector and the resource semantic embedding vector to generate an interaction feature matrix; The basic similarity features and the interaction feature matrix are subjected to joint attention embedding processing to generate a joint embedding feature vector. The joint embedded feature vector is subjected to matching degree scoring through the fully connected classification layer of the causal inference network model to generate the initial matching score vector.

6. The library cross-modal retrieval method based on semantic understanding according to claim 1, characterized in that, The root factor graph retrieval process performed on the multimodal resource database based on the resource correlation matrix to generate a target retrieval result set includes: The resource correlation matrix is ​​subjected to threshold filtering to select a list of candidate resource identifiers with correlation scores higher than a preset threshold. For each candidate resource identifier in the candidate resource identifier list, extract the key causal node sequence in its corresponding resource semantic graph; The pre-trained path reasoning model is invoked to perform semantic coherence verification on the key causal node sequence, generating a node sequence that passes the verification. The verified node sequence is matched with the query semantic graph to generate the target retrieval result set, which contains library resource information that is most closely related to the semantic causal relationship of the user's query text.

7. A cross-modal library retrieval method based on semantic understanding according to claim 6, characterized in that, The step of calling the pre-trained path reasoning model to perform semantic coherence verification on the key causal node sequence and generating a node sequence that passes the verification includes: The key causal node sequence is input into the sequence encoder of the path reasoning model for node position encoding processing to generate an encoded node sequence; The conditional random field layer of the path reasoning model is invoked to calculate the label transition probability of the encoded node sequence, generating a node coherence probability distribution; Based on the node coherence probability distribution, nodes with probabilities lower than a preset threshold are removed from the key causal node sequence to obtain the verified node sequence.

8. A cross-modal library retrieval method based on semantic understanding according to claim 6, characterized in that, The step of performing subgraph matching processing between the verified node sequence and the query semantic graph to generate the target retrieval result set includes: The verification process involves subgraph reconstruction of the node sequence to generate a candidate resource subgraph. Calculate the structural similarity score and semantic similarity score between the candidate resource subgraph and the query semantic graph; The structural similarity score and the semantic similarity score are weighted and summed to obtain a comprehensive matching score; Based on the comprehensive matching score, the resources in the candidate resource identifier list are sorted, and the top N resources are selected as the target retrieval result set.

9. A cross-modal library retrieval method based on semantic understanding according to claim 3, characterized in that, The method for constructing the pre-trained causal inference network model includes: Obtain a cross-modal sample dataset, with each set of sample data including sample query text, sample multimodal resources, and corresponding sample causal association labels; The multidimensional semantic graph construction process is performed on each of the sample query text and each resource in the sample multimodal resources to obtain the sample query semantic graph and the sample resource semantic graph. The graph embedding encoding process is performed on the sample query semantic graph and the sample resource semantic graph respectively to obtain the sample query embedding vector and the sample resource embedding vector. The sample query embedding vector and the sample resource embedding vector are input into the initial causal inference network, which includes a cross-modal contrastive learning layer and a causal path discovery layer. The cross-modal contrastive learning layer outputs an initial sample matching score vector, and the causal path discovery layer outputs a sample resource correlation prediction matrix. Based on the sample resource correlation prediction matrix and the sample causal correlation label, the joint loss function value is calculated, and the initial causal inference network is backpropagated and the parameters are iteratively updated until convergence, thus obtaining the pre-trained causal inference network model.

10. A cross-modal library retrieval method based on semantic understanding according to claim 1, characterized in that, The method further includes: Parse the resource semantic graph corresponding to each search result in the target search result set, and extract the modality type node distribution in the resource semantic graph; Based on the distribution of nodes of the modal type, a cross-modal retrieval feedback signal is generated. The cross-modal retrieval feedback signal is used to indicate the contribution ratio of each modal resource to the user query in the retrieval results. The cross-modal retrieval feedback signal and the resource correlation matrix are input into the policy optimization network model for reinforcement learning policy update processing to generate updated causal inference network model parameters, specifically including: The cross-modal retrieval feedback signal is encoded into a reward value vector; The reward value vector and the resource correlation matrix are concatenated to generate a state feature vector. The policy gradient calculation layer of the network model is invoked to perform gradient descent calculation on the state feature vector to generate model parameter update amounts. The weight parameters of the causal inference network model are adjusted by batch normalization using the model parameter update amount to generate the updated causal inference network model parameters.

Images