A Multimodal Document Retrieval Method and Device Based on Cross-Modal Mutual Attention Mechanism

Abstract

This application relates to the field of document retrieval technology, and particularly to a multimodal document retrieval method and apparatus based on a cross-modal mutual attention mechanism. The method includes: modeling a multimodal representation of a document; obtaining a target document-perceived multimodal document representation based on a multimodal mutual attention mechanism; fusing the document's self-attention vectorized representation and multimodal enhanced vectorized representation to obtain a unified multimodal enhanced representation of the document; calculating and ranking the relevance scores between the target document and at least one candidate document; and retrieving relevant documents. This application's embodiments, based on a cross-modal mutual attention mechanism, can obtain matching documents by acquiring a unified multimodal enhanced representation of the document and calculating relevance scores, thereby fully utilizing the multimodal information of the document, enhancing the relevance between different modalities of the document, and thus improving the matching degree of the document retrieval results, making the retrieval results more accurate and reliable.

Description

Technical Field

[0001] This application relates to the field of document retrieval technology, and in particular to a multimodal document retrieval method and apparatus based on a cross-modal mutual attention mechanism. Background Technology

[0002] Document retrieval is an important research area in natural language processing. It refers to retrieving the most relevant document from a large number of documents, and it forms the basis for the research of many downstream tasks.

[0003] With the continuous development of Internet technology, the information contained in documents is gradually increasing. Documents contain a variety of information, including text and images. Each modality can be modeled separately to achieve multimodal document retrieval.

[0004] However, in related technologies, the correlation between different modalities is ignored when modeling each modality, and the multimodal content interaction within a document and between different documents is not considered. This makes it impossible to effectively utilize multimodal information within a document, which affects the document retrieval effect and reduces the accuracy and relevance of document retrieval, and urgently needs to be solved. Summary of the Invention

[0005] This application provides a multimodal document retrieval method and apparatus based on a cross-modal mutual attention mechanism to solve the problems in related technologies, such as ignoring the correlation between different modalities during modal modeling, not considering the multimodal content interaction within a document and between different documents, failing to effectively utilize multimodal information within a document, affecting the document retrieval effect, and reducing the accuracy and relevance of document retrieval.

[0006] The first aspect of this application provides a multimodal document retrieval method based on a cross-modal mutual attention mechanism, comprising the following steps: modeling the multimodal representation of a document, obtaining a first vectorized representation sequence of the text modality and a second vectorized representation sequence of the image modality of the document; jointly modeling the text modality and the image modality based on the first and second vectorized representation sequences, obtaining a multimodal vectorized representation of the text and a multimodal vectorized representation of the image, to model the interaction between different modalities of the document based on a multimodal mutual attention mechanism, and modeling a multimodal enhanced vectorized representation of the document; modeling the correlation between the same and different modalities of the document and the target document based on the mutual attention mechanism, obtaining a multimodal document representation perceived by the target document; modeling the autocorrelation within the document modality, fusing the self-attention vectorized representation of the document and the multimodal enhanced vectorized representation, obtaining a multimodal enhanced unified representation of the document; calculating the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation, and ranking the at least one candidate document based on the relevance score to retrieve relevant documents.

[0007] Optionally, in one embodiment of this application, before modeling the multimodal representation of the document, the method further includes: preprocessing the text data of the document, filtering out text that does not meet preset conditions, and converting the filtered text into a text sequence to obtain text data for modeling; preprocessing the image data of the document, converting the image into a pixel sequence, and normalizing the mean and variance of the image to obtain image data for modeling.

[0008] Optionally, in one embodiment of this application, the multimodal representation of the modeling document, which obtains a first vectorized representation sequence of the text modality and a second vectorized representation sequence of the image modality, includes: encoding word vector representations of the text in the document based on a word vector embedding matrix to obtain the first vectorized representation sequence of the document; generating image candidate boxes for the images of the document based on a pre-trained Mask-RCNN network; and extracting features from the image candidate boxes based on a ResNet-50 network to obtain the second vectorized representation sequence of the document.

[0009] Optionally, in one embodiment of this application, the step of jointly modeling the text modality and the image modality based on the first and second vectorized representation sequences to obtain multimodal vectorized representations of document text and document images, respectively, to model the interaction between different modalities of the document based on a multimodal mutual attention mechanism, and to model the multimodal enhanced vectorized representation of the document, includes: jointly modeling the text modality and the image modality based on VilBERT (Vision-and-LanguageBERT) to obtain multimodal vectorized representations of the text and images; modeling the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and modeling the interrelationship between different modalities within the document based on a correlation matrix to obtain the interaction between different modalities within the document, and modeling the multimodal enhanced vectorized representation of the document.

[0010] Optionally, in one embodiment of this application, the step of modeling the correlation between the same and different modalities of the document and the target document based on the mutual attention mechanism to obtain the multimodal document representation perceived by the target document includes: based on the mutual attention mechanism, interacting with the same modal content of the document and the target document to model the correlation between the same modalities of the document and the target document to obtain the cross-document same modal document augmented representation perceived by the target document; and based on the cross-modal mutual attention mechanism between documents, interacting with the different modal content of the document and the target document to model the correlation between multiple document modalities to obtain the multimodal document representation perceived by the target document.

[0011] Optionally, in one embodiment of this application, the process of modeling the autocorrelation within document modalities and fusing the self-attention vectorized representation of the document with the vectorized representation of the multimodal enhancement to obtain a unified multimodal enhancement representation of the document includes: modeling the autocorrelation within document modalities based on a multi-head self-attention mechanism to obtain the self-attention vectorized representation of the document; fusing the self-attention vectorized representation of the document with the vectorized representations of various multimodal enhancements of the document to obtain a unified multimodal enhancement representation of the document modalities; and fusing the unified multimodal enhancement representations of different modalities of the document to obtain the unified multimodal enhancement representation.

[0012] Optionally, in one embodiment of this application, the step of calculating the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation, and sorting the at least one candidate document based on the relevance score to retrieve the relevant document includes: calculating the relevance score between the target document and the at least one candidate document based on the multimodal enhanced unified representation; and sorting the at least one candidate document based on the relevance score of each candidate document to determine the relevant document.

[0013] A second aspect of this application provides a multimodal document retrieval device based on a cross-modal mutual attention mechanism, comprising: a first modeling module, configured to model the multimodal representation of a document, obtaining a first vectorized representation sequence of the text modality and a second vectorized representation sequence of the image modality of the document; and a second modeling module, configured to jointly model the text modality and the image modality based on the first and second vectorized representation sequences, obtaining a multimodal vectorized representation of the text and a multimodal vectorized representation of the image, thereby modeling the interaction between different modalities of the document based on a multimodal mutual attention mechanism, and modeling the multimodality of the document. The system includes: an enhanced vectorized representation; a third modeling module, used to model the correlation between the same and different modalities of the document and the target document based on a mutual attention mechanism, to obtain a multimodal document representation perceived by the target document; a fourth modeling module, used to model the autocorrelation within the document modality, and to fuse the self-attention vectorized representation of the document with the enhanced multimodal vectorized representation, to obtain a unified multimodal representation of the document; and a retrieval module, used to calculate the relevance score between the target document and at least one candidate document based on the unified multimodal representation, and to rank at least one candidate document based on the relevance score, thereby retrieving relevant documents.

[0014] Optionally, in one embodiment of this application, it further includes: a first conversion module, configured to preprocess the text data of the document before modeling the multimodal representation of the document, filter out text that does not meet preset conditions, and convert the filtered text into a text sequence to obtain text data for modeling; and a second conversion module, configured to preprocess the image data of the document, convert the image into a pixel sequence, and normalize the mean and variance of the image to obtain image data for modeling.

[0015] Optionally, in one embodiment of this application, the first modeling module includes: an encoding unit, configured to encode word vector representations of text in the document based on a word vector embedding matrix to obtain a first vectorized representation sequence of the document; a generation unit, configured to generate image candidate boxes for images in the document based on a pre-trained Mask-RCNN network; and an extraction unit, configured to extract features from the image candidate boxes based on a ResNet-50 network to obtain a second vectorized representation sequence of the document.

[0016] Optionally, in one embodiment of this application, the second modeling module includes: a first modeling unit, configured to jointly model the text modality and the image modality based on VilBERT to obtain a multimodal vectorized representation of the text and a multimodal vectorized representation of the image; and a second modeling unit, configured to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and to model the mutual relationship between different modalities within the document based on the relevance matrix to obtain the interaction between different modalities within the document, and to model the multimodal enhanced vectorized representation of the document.

[0017] Optionally, in one embodiment of this application, the third modeling module includes: a first interaction unit, configured to interact with the same-modal content of the document and the target document based on the mutual attention mechanism, model the correlation between the same modalities of the document and the target document, and obtain a cross-document same-modal document augmentation representation perceived by the target document; and a second interaction unit, configured to interact with different-modal content of the document and the target document based on the cross-modal mutual attention mechanism between documents, model the correlation between multiple document modalities, and obtain a multimodal document representation perceived by the target document.

[0018] Optionally, in one embodiment of this application, the fourth modeling module includes: a third modeling unit, configured to model the autocorrelation within a document modality based on a multi-head self-attention mechanism to obtain a self-attention vectorized representation of the document; a first fusion unit, configured to fuse the self-attention vectorized representation of the document and vectorized representations of various multimodal enhancements of the document to obtain a unified multimodal enhancement representation of the document modality; and a second fusion unit, configured to fuse the unified multimodal enhancement representations of different modalities of the document to obtain the unified multimodal enhancement representation.

[0019] Optionally, in one embodiment of this application, the retrieval module includes: a calculation unit, configured to calculate the relevance score between the target document and the at least one candidate document based on the multimodal enhanced unified representation; and a sorting unit, configured to sort the at least one candidate document based on the relevance score of each candidate document to determine the relevant documents.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the multimodal document retrieval method based on a cross-modal mutual attention mechanism as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal document retrieval method based on a cross-modal mutual attention mechanism as described above.

[0022] This application's embodiments can model a multimodal representation of a document, specifically a target document-aware multimodal representation based on a cross-modal mutual attention mechanism between documents. This yields a unified representation with enhanced multimodal representation, and a relevance score is calculated to retrieve matching documents. This fully utilizes the multimodal information of the document, enhances the relevance between different modalities, and improves the matching degree of the document retrieval results, making the retrieval results more accurate and reliable. Therefore, it solves the problems in related technologies where the modeling of each modality ignores the relevance between different modalities, fails to consider the multimodal content interaction within a document and between different documents, and cannot effectively utilize the multimodal information within the document, thus affecting the document retrieval effect and reducing the accuracy and relevance of document retrieval.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 This is a flowchart illustrating a multimodal document retrieval method based on a cross-modal mutual attention mechanism, according to an embodiment of this application.

[0026] Figure 2 This is a schematic diagram illustrating a document retrieval process based on a cross-modal mutual attention mechanism according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating the principle of document retrieval based on a cross-modal mutual attention mechanism according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the structure of a multimodal document retrieval device based on a cross-modal mutual attention mechanism according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0031] The following description, with reference to the accompanying drawings, describes a multimodal document retrieval method and apparatus based on a cross-modal mutual attention mechanism according to embodiments of this application. Addressing the issues raised in the background art, where modal modeling neglects the correlation between different modalities and fails to consider multimodal content interaction within and between documents, the method cannot effectively utilize multimodal information within a document, thus affecting document retrieval performance and reducing the accuracy and relevance of document retrieval. This application provides a multimodal document retrieval method based on a cross-modal mutual attention mechanism. By modeling a document's multimodal representation, and then modeling a target document-perceived multimodal representation based on a cross-modal mutual attention mechanism, a unified representation with enhanced multimodal representation is obtained. A relevance score is calculated to retrieve matching documents, thereby fully utilizing the document's multimodal information, enhancing the correlation between document modalities, and improving the matching degree of document retrieval results, making the retrieval results more accurate and reliable. This solves the problems in related technologies, such as neglecting the correlation between different modalities during modal modeling, failing to consider the multimodal content interaction within a document and between different documents, failing to effectively utilize multimodal information within a document, affecting document retrieval performance, and reducing the accuracy and relevance of document retrieval.

[0032] Specifically, Figure 1 This is a flowchart illustrating a multimodal document retrieval method based on a cross-modal mutual attention mechanism, provided in an embodiment of this application.

[0033] like Figure 1 As shown, this multimodal document retrieval method based on cross-modal mutual attention mechanism includes the following steps:

[0034] In step S101, the multimodal representation of the document is modeled, and the first vectorized representation sequence of the text mode and the second vectorized representation sequence of the image mode are obtained respectively.

[0035] It is understood that in the embodiments of this application, the first vectorized representation sequence can be obtained from the text modality of the document, the second vectorized representation sequence can be obtained from the image modality of the document, and the modeling result of the document multimodal representation can include the text content and image content of the document.

[0036] The embodiments of this application can model the multimodal representation of a document, obtaining a first vectorized representation sequence of the document's text modality and a second vectorized representation sequence of the image modality, thereby achieving separate modeling for the text modality and image modality within the document.

[0037] Optionally, in one embodiment of this application, before modeling the multimodal representation of the document, the method further includes: preprocessing the text data of the document, filtering out text that does not meet preset conditions, and converting the filtered text into a text sequence to obtain text data for modeling; preprocessing the image data of the document, converting the image into a pixel sequence, and normalizing the mean and variance of the image to obtain image data for modeling.

[0038] It is understood that in the embodiments of this application, the processed text sequence can be obtained by preprocessing the text data, and the processed pixel sequence can be obtained by preprocessing the image data, so as to be used for modeling the multimodal representation of the document in the above steps.

[0039] It should be noted that the preset conditions are set by those skilled in the art based on the actual situation, and no specific limitations are made here.

[0040] Optionally, in one embodiment of this application, modeling the multimodal representation of a document to obtain a first vectorized representation sequence of the document's text modality and a second vectorized representation sequence of the document's image modality includes: encoding word vector representations of the text in the document based on a word vector embedding matrix to obtain the first vectorized representation sequence of the document; generating image candidate boxes for the document's images based on a pre-trained Mask-RCNN network; and extracting features of the image candidate boxes based on a ResNet-50 network to obtain the second vectorized representation sequence of the document.

[0041] For example, the word vector representation of the text can be encoded using a word embedding matrix to obtain the text vectorization representation of the document. A pre-trained Mask-RCNN network can be used to generate candidate boxes for all images in the document. The generated image candidate boxes can be modeled using a ResNet50 network to obtain the image vectorization representation of the document.

[0042] In actual execution, for a given document, it can be represented by the text sequence w = [w1, w2, ..., w...]. M [] represents the text of the document, where M is the length of the text sequence. Each word w in the text sequence i Through character embedding matrix Mapping to a vector sequence Among them, V w For vocabulary size, D w The resulting vector sequence is the embedding dimension. This refers to the vectorized representation of the document text, specifically the first sequence of vectorized representations of the document.

[0043] Furthermore, each image in a given document can be viewed as a two-dimensional array of pixels, p = [p1, p2, ..., p...]. M[M], where M is the length of the image pixel sequence. The Mask-RCNN network generates candidate boxes for each object in the image, representing the important parts of the image in the form of candidate boxes. For each image, the Mask-RCNN network generates K candidate boxes, and the sequence of candidate boxes can be represented as p = [p1, p2, ..., p...]. K [K], where K is the number of ROIs. Then, a ResNet50 network is used to model the generated image candidate boxes, extracting their features to obtain a vectorized representation of the image candidate box sequence. Depend on Obtain vector sequence This is the vectorized representation of the document's images, which is the second vectorized representation sequence of the document.

[0044] The embodiments of this application can encode the word vector representation of the text in the document to obtain the first vectorized representation sequence of the document, and generate image candidate boxes for the images in the document, extract the features of the image candidate boxes, and obtain the second vectorized representation sequence of the document, thereby realizing the modeling process of vectorized representation of each modality of the document.

[0045] In step S102, based on the first vectorized representation sequence and the second vectorized representation sequence, the text modality and the image modality are jointly modeled to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image, respectively. This is to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and to model the multimodal enhanced vectorized representation of the document.

[0046] It is understood that, in the embodiments of this application, joint modeling can refer to cross-modal modeling of the document's text sequence and image sequence to obtain multimodal vectorized representations of the document's text modality and image modality. The modeling of the multimodal mutual attention mechanism can be performed within the document, interacting with inputs from different modalities to ultimately obtain the document's multimodal enhanced vectorized representation, namely, the multimodal enhanced text matrix and the multimodal enhanced image matrix.

[0047] Optionally, in one embodiment of this application, the text modality and image modality are jointly modeled based on the first vectorized representation sequence and the second vectorized representation sequence to obtain the multimodal vectorized representation of the document text and the multimodal vectorized representation of the document image, respectively. This is to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and to model the multimodal enhanced vectorized representation of the document. This includes: jointly modeling the text modality and image modality based on VilBERT to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image; modeling the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and modeling the mutual relationship between different modalities within the document based on the correlation matrix to obtain the interaction between different modalities within the document, thereby modeling the multimodal enhanced vectorized representation of the document.

[0048] It is understood that the ViLBERT network model in this application embodiment can be used for multimodal joint modeling. Based on the Transformer layer of the multimodal mutual attention mechanism, it can interact for different modalities at different depths to obtain multimodal enhanced document text and document image representations.

[0049] For example, the interaction between different modalities of a document can be modeled based on the multimodal mutual attention mechanism within the document. The normalized relevance score is calculated using the normalized exponential function based on the relevance matrix and used as the normalized weight coefficient. The document representation is then weighted based on the normalized relevance score to obtain the multimodal enhanced representation of the document.

[0050] Specifically, the text sequence and image sequence of the document in the above steps can be vectorized. As a common input to ViLBERT, it is also fed into the ViLBERT network model:

[0051]

[0052] income Vectorized representation of the text sequence of a document. The document's image sequence is vectorized. Then, the vectorized representation sequence of the document's text modality is concatenated to obtain the document's text representation matrix, i.e.:

[0053]

[0054] in, The text representation matrix of the document. Vectorized representation of the text sequence of a document. M w Let E be the length of the text sequence, and E be the dimension of the sequence vectorization representation output by the model. Concatenating the vectorization representation sequences of the document's image modalities yields the document's image representation matrix, i.e.:

[0055]

[0056] in, The image representation matrix of the document. Vectorize the image sequence representation of the document. M p Let E be the length of the document image sequence, and let E be the dimension of the sequence vectorization representation output by the model. Thus, we obtain the multimodal vectorization representation of the document text and the multimodal vectorization representation of the document image, respectively.

[0057] Furthermore, it can be based on the correlation matrix C XY The correlation matrix C represents the correlation between different modalities of content, where X and Y represent different modalities of the document. XY Elements at each position The correlation coefficient represents the correlation between the vector at position i in document modality X and the vector at position j in document modality Y, characterizing the degree of association between the two vectors. The resulting correlation matrix C is calculated. XY It can characterize the correlation between elements at various positions between document modalities X and Y. For the representation matrix of the document text modality and the representation matrix of the image modality, the correlation matrix is ​​specifically as follows:

[0058] C WP =(S W ) T W WP S P ,

[0059] C PW =(S P ) T W PW S W ,

[0060] Among them, C WP S is the correlation matrix between the text modality and the image modality of the document. W W is the representation matrix of the document text modality. WP S is the weight matrix for the text modality and image modality of the document. P C is the representation matrix of the image modalities. PW W represents the correlation matrix between the image modality and the text modality of the document. PW W represents the weight matrix for the image and text modalities of the document. WP W PW We can perform weighted summations of the vector inner products at corresponding positions, C WP ∈R P×E C PW ∈R E×E E represents the dimension of the sequence vectorization representation of the model output. This is determined by the normalized exponential function acting on the correlation matrix C.WP The normalized relevance score matrix between the document text modality and the image modality is calculated as follows:

[0061] α WP =Softmax(C WP ),

[0062] Where, α WP C is the normalized relevance score matrix for the document text modality and the image modality. WP Let α be the correlation matrix between the text modality and the image modality of the document. WP ∈R E×E E represents the dimension of the sequence vectorization representation output by the model. The correlation matrix C between the image and text modalities of the document is determined by the normalized exponential function. PW The normalized correlation score matrix between the document image modality and the text modality is calculated as follows:

[0063] α PW =Softmax(C PW ),

[0064] Where, α PW C is the normalized relevance score matrix for document image modal and text modal. PW Let α be the correlation matrix between the image modality and the text modality of the document. PW ∈R E×E E represents the dimension of the sequence vectorization representation output by the model. Based on the obtained normalized relevance score, the document representation can be weighted to calculate the cross-modal enhanced text modality representation, i.e.:

[0065]

[0066] Among them, S W For cross-modal enhanced text modality representation, Let α be the text representation matrix of the document. WP This is the normalized relevance score matrix for document text modal and image modal. This refers to the enhanced representation of the document text modality, M. w Let E be the length of the text sequence, and E be the dimension of the sequence vectorization representation output by the model. The cross-modal augmented image modal representation is calculated as follows:

[0067]

[0068] Among them, S P For cross-modal enhanced image modal representation, Let α be the image modal representation matrix of the document. PW This is the normalized correlation score matrix between the document image modality and the text modality. M p E represents the length of the document image sequence, and E represents the dimension of the sequence vectorization representation output by the model.

[0069] In step S103, based on the mutual attention mechanism, the correlation between the same and different modalities of the document and the target document is modeled to obtain the multimodal document representation perceived by the target document.

[0070] It is understood that the cross-modal mutual attention mechanism in the embodiments of this application is executed between documents. It can obtain the multimodal document representation perceived by the target document based on the correlation between the same and different modalities of the current document and the target document, respectively, for the vectorized representation of each modality within the current document and the vectorized representation of each modality within the target document obtained by the above steps.

[0071] This application's embodiments can model the correlation between the same and different modalities of a document and a target document based on a mutual attention mechanism, obtaining a multimodal document representation perceived by the target document. By modeling the correlation between the current document and the target document, further matching relationships between documents can be established.

[0072] Optionally, in one embodiment of this application, modeling the correlation between the same and different modalities of a document and a target document based on a mutual attention mechanism to obtain a multimodal document representation perceived by the target document includes: based on the mutual attention mechanism, interacting with the same modal content of the document and the target document to model the correlation between the same modalities of the document and the target document to obtain a cross-document same modal document augmentation representation perceived by the target document; and based on the cross-modal mutual attention mechanism between documents, interacting with the different modal content of the document and the target document to model the correlation between multiple document modalities to obtain a multimodal document representation perceived by the target document.

[0073] For example, based on a mutual attention mechanism, interaction can be performed on the same modal content of the current document and the target document. A normalized relevance score matrix is ​​calculated using a normalized exponential function from the relevance matrix, serving as the normalized weight coefficients. This models the relevance between the same modalities of the current and target documents, resulting in a cross-document same-modal document augmentation representation perceived by the target document. Simultaneously, based on a cross-modal mutual attention mechanism between documents, interaction can be performed on different modal content of the current and target documents. A normalized relevance score matrix is ​​calculated using a normalized exponential function from the relevance matrix, serving as the normalized weight coefficients. The document representation is weighted based on this normalized relevance score matrix, modeling the cross-document cross-modal interaction between a given modality of the current document and another modality of the target document, resulting in a cross-modal document representation perceived by the target document.

[0074] Specifically, it can be based on the correlation matrix C ABRelevance matrix C represents the correlation between the same modal content in different documents, with A and B representing the current document and the target document, respectively. AB Elements at each position The correlation coefficient represents the vector at position i in modality A and the vector at position j in modality B, characterizing the degree of correlation between the two vectors. The calculated correlation matrix C AB The correlation between elements at various positions in the corresponding modalities of documents A and B is characterized as follows:

[0075] C AB =(S A ) T W AB S B ,

[0076] C BA =(S B ) T W BA S A ,

[0077] Among them, C AB S is the correlation score matrix between the document text modality and the document image modality. A Let W be the representation matrix of document A. AB S is the weight matrix for the document text modality and the document image modality. B Let C be the representation matrix of target document B. BA W is the correlation score matrix between the image modality and the text modality of the document. BA W represents the weight matrix for the image and text modalities of the document. AB W BA We can calculate C by weighting the vector inner product at the corresponding positions. AB ∈R E×E and C BA ∈R E×E The correlation between the document's text modality W and image modality P is represented by E, where E is the dimension of the sequence vectorization representation output by the model. The correlation matrix C between the document's text and image modalities can be calculated by applying an exponential function to the correlation matrix. AB The normalized relevance score matrix between the document text modality and the image modality is calculated, which gives:

[0078] α AB =Softmax(C AB ),

[0079] Where, α AB C is the normalized relevance score matrix for the document text modality and the image modality. AB Let α be the correlation score matrix between the document text modality and the document image modality. AB∈R E×E E represents the dimension of the sequence vectorization representation output by the model. This can be expressed by applying a normalized exponential function to the correlation matrix C between the image and text modalities of the document. BA The normalized correlation score matrix between the document image modality and the text modality is calculated, i.e.:

[0080] α BA =Softmax(C BA ),

[0081] Where, α BA C is the normalized relevance score matrix for document image modal and text modal. BA Let α be the relevance score matrix between the image modality and the text modality of the document. BA ∈R E×E E represents the dimension of the sequence vectorization representation output by the model. For a given modality matrix representation of the current document A, the cross-modal enhanced document representation is calculated by weighting it based on the normalized relevance score matrix of the same modality of the target document B, i.e.:

[0082]

[0083] Among them, S A For the target document-aware augmented representation of the current document, Given a matrix representation of the modality of the current document A, α AB This is the normalized relevance score matrix for document text modal and image modal. X represents the current interaction modality, and E represents the dimension of the sequence vectorization representation output by the model.

[0084] Meanwhile, based on the correlation matrix C XY The correlation matrix represents the correlation between different modalities of content in different documents, where X and Y represent a given modality of the current document and another modality of the target document, respectively. XY Elements at each position The correlation coefficient represents the vector at position i in a given modality of the current document and the vector at position j in another modality of the target document. It characterizes the degree of correlation between the two vectors. The calculated correlation matrix C XY The correlation matrix C represents the correlation between elements at various positions in a given modality of the current document and another modality of the target document. XY for:

[0085] C XY =(S X ) T W XY S Y ,

[0086] Among them, C XYS is the correlation matrix between the text modality and the image modality of the document. X Given the representation matrix of modality X in the current document, W XY S is the weight matrix for the text modality and image modality of the document. Y Let C be the representation matrix of another modality Y of the target document. XY ∈R E×E E is the dimension of the sequence vectorization representation of the model output, and W is the dimension of the sequence vectorization representation. XY The dot product of vectors at corresponding positions can be weighted. For the correlation matrix C between the text modality and the image modality of a document... XY The normalized exponential function can be applied to the correlation matrix C. XY The normalized correlation score matrix between the two modes is calculated as follows:

[0087] α XY =Softmax(C XY ),

[0088] Where, α XY C is the normalized relevance score matrix for another modality of the target document. XY Let α be the correlation matrix between the text modality and the image modality of the document. XY ∈R E×E E represents the dimension of the sequence vectorization representation output by the model. For a given modality matrix representation of the current document, the cross-modal enhanced document representation is calculated by weighting it based on its normalized relevance score matrix with another modality of the target document, i.e.:

[0089]

[0090] Among them, S X To achieve cross-modal enhanced document representation, Given a matrix representation of the modality of the current document, α XY This is a normalized relevance score matrix for another modality of the target document. X represents the current interaction modality, and E is the dimension of the sequence vectorization representation of the model output.

[0091] In step S104, the autocorrelation within the document modality is modeled, and the self-attention vectorized representation and the multimodal enhancement vectorized representation of the document are fused to obtain the unified multimodal enhancement representation of the document.

[0092] It is understood that the unified representation of document multimodal enhancement in the embodiments of this application can realize the unified expression of document text and document images. The unified representation of document multimodal enhancement can be obtained by fusing the enhanced vectorized representations of multiple modalities within the document, thereby further realizing the basis for measuring document relevance and improving the matching degree of search results.

[0093] Optionally, in one embodiment of this application, modeling the autocorrelation within document modalities and fusing the self-attention vectorized representation of the document with the vectorized representation of multimodal enhancements to obtain a unified multimodal enhancement representation of the document includes: modeling the autocorrelation within document modalities based on a multi-head self-attention mechanism to obtain a self-attention vectorized representation of the document; fusing the self-attention vectorized representation of the document with the vectorized representations of various multimodal enhancements of the document to obtain a unified multimodal enhancement representation of the document modalities; and fusing the unified multimodal enhancement representations of different modalities of the document to obtain a unified multimodal enhancement representation.

[0094] For example, autocorrelation within a document modality can be modeled based on a multi-head self-attention mechanism to obtain a self-attention vectorized representation of the document modality. A normalized relevance score is calculated using a normalized exponential function, and the document representation is weighted to model the autocorrelation attention interactions of the current document in a given modality, resulting in a cross-modal document representation perceived by the target document. Based on an attention pooling mechanism, the document's autocorrelation attention-enhanced vectorized representation matrix and the vectorized representation matrices of multiple multimodal enhancements are fused to obtain a unified multimodal enhancement representation of the document modality. The multimodal mutual attention enhancement representation matrices are then fused to obtain a multimodal enhancement attention fusion vector of the document modality. Finally, a unified representation of the document is obtained by fusing the multimodal enhancement attention fusion vectors of different modalities of the document based on a feedforward neural network.

[0095] In practice, multi-head self-attention mechanisms can calculate the importance of each element in the sequence. For the modal representation matrix of a document, the autocorrelation matrix C can be used. X Characterizing the autocorrelation within document modalities, where X represents different document modalities, and C... X The element at each position can represent the correlation between elements at different positions in different modalities of the document, as follows:

[0096] C X =(S X ) T W X S X ,

[0097] Among them, C X Let S be the representation matrix of document X modality. X W is the representation matrix of document X modality. X W is the weight matrix for document X modality. X C is a learnable variable, acquired during training. X The value at each position The correlation coefficient represents the relationship between the vector at position i and the vector at position j, characterizing the degree of association between the two vectors. For a given modality of a document, the correlation matrix C...X The normalized exponential function can be applied to the correlation matrix C. X The normalized relevance scores between the document text modality and the image modality are calculated above, i.e.:

[0098] α X =Softmax(C X ),

[0099] Where, α X C represents the autocorrelation-normalized relevance score of a document. X Let X be the representation matrix of the modality of document X. For a given modality representation of the current document, the matrix representation is based on its autocorrelation-normalized relevance score matrix α. XY After weighting, the cross-modal enhanced document representation is calculated, namely:

[0100]

[0101] Among them, S X For cross-modal enhanced document representation, For a given modality of the previous document, α X This is the autocorrelation-normalized correlation score matrix. X represents the current interaction modality, and E represents the dimension of the sequence vectorization representation output by the model.

[0102] For each modality of the document, four multimodal mutual attention enhancement representations based on attention mechanisms are calculated: an autocorrelation attention enhancement representation based on self-attention, a cross-modal enhancement representation based on cross-modal mutual attention, a cross-document mutual attention enhancement representation based on cross-document mutual attention, and a cross-document cross-modal mutual attention enhancement representation based on cross-document cross-modal mutual attention. For a document's modality X, its original vectorized representation matrix is ​​S0, and its multimodal mutual attention enhancement representation matrices are: the autocorrelation attention enhancement representation matrix based on self-attention... self Cross-modal enhanced representation matrix S based on cross-modal mutual attention mechanism multimodal A cross-document mutual attention enhancement representation matrix S based on a cross-document mutual attention mechanism for target document awareness. multdoc A target document-aware cross-document cross-modal mutual attention enhancement representation matrix S. multidoc-multimodal Based on the attention pooling mechanism, these are fused to obtain a multimodal enhanced attention fusion vector, i.e.:

[0103]

[0104] Where k is the attention fusion vector for multimodal enhancement. Let W1 be the attention query matrix, W2 be the weight matrix for weighting different multimodal mutual attention enhancement representation matrices, and W4 be the hyperbolic tangent activation function. The attention fusion vectors for text modality W and image modality P are calculated separately and then concatenated to obtain a unified joint vector, i.e.:

[0105] k c =[k W ;k P ]

[0106] Where, k c Let k be the joint vector. W k is the attention fusion vector for multimodal enhancement of text modality W. P This is the attention fusion vector for multimodal enhancement of image modality P. The resulting joint vector is then fed into a feedforward neural network to further fuse representation features from different document modalities. That is:

[0107] k = FFNN(k c )

[0108] Where k is the unified document representation, k c The joint vector, FFNN represents a feedforward neural network, and the final output is a multimodal enhanced unified representation of the document.

[0109] In step S105, based on the multimodal enhanced unified representation, the relevance score between the target document and at least one candidate document is calculated, and the at least one candidate document is sorted based on the relevance score to retrieve the relevant document.

[0110] It is understood that the matching score between the target document and the current document in this application embodiment can represent the degree of correlation between the two documents. Based on the multimodal enhanced unified representation of the document obtained in the above steps, the relevance score can be calculated, and the document with the strongest relevance can be selected as the matching result output according to the ranking result of the score. In this way, the full utilization of the multimodal information of the document can be realized, thereby improving the matching degree of the document retrieval results and making the retrieval results more accurate and reliable.

[0111] Optionally, in one embodiment of this application, the relevance score between the target document and at least one candidate document is calculated based on the multimodal enhanced unified representation, and the at least one candidate document is sorted based on the relevance score to retrieve the relevant document, including: calculating the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation; sorting the at least one candidate document based on the relevance score of each candidate document to determine the relevant document.

[0112] In actual execution, for the current document A and the target document B that need to be matched, the unified representation k of document A is calculated respectively. A The unified representation of B documents k B Calculate the inner product of the image modalities in document A and document B, and add the two inner integrals to obtain the final matching score.

[0113]

[0114] in, k is the matching score between the current document and the target document. A k is the unified representation of document A. B This provides a unified representation for B-documents. For each candidate document, multiple negative samples are randomly sampled, and their matching scores with the target document are calculated. The BPR (Bayesian Personalized Ranking) loss is then used for model learning.

[0115]

[0116] Where L is the loss function and H is the size of the training samples. Let be the matching score of the i-th positive sample. Let φ be the matching score of the i-th negative sample, and let φ be the Sigmoid function. Then, the loss function L is optimized and the model is trained by the RMSProp optimizer to obtain the optimal parameters for document modeling.

[0117] The working content of the embodiments of this application will be described in detail below with a specific example.

[0118] like Figure 2-3 As shown, text can be converted into a text sequence, text can be filtered, and images can be converted into pixel sequences. The mean and variance are normalized. Based on the document vectorization encoder, the vectorized representation of each modality of the document is modeled. Based on the cross-modal mutual attention mechanism within the document, the multimodal representation of the document is modeled. Based on the cross-modal mutual attention mechanism between documents, the multimodal representation of the target document is modeled. The multimodal enhanced vectorized representation of the document is fused to obtain a unified representation of the document's multimodal enhancement. Based on the document modeling representation result of multimodal enhancement, the most relevant matching document is retrieved.

[0119] The multimodal document retrieval method based on cross-modal mutual attention mechanism proposed in this application can obtain a unified representation of enhanced multimodal document representation by modeling multimodal representations of documents and modeling the target document's perceived multimodal representation based on the cross-modal mutual attention mechanism between documents. A relevance score is then calculated to retrieve matching documents, thereby fully utilizing the multimodal information of the document, enhancing the relevance between different modalities, and improving the matching degree of the document retrieval results, making the retrieval results more accurate and reliable. This solves the problems in related technologies where the modeling of each modality ignores the relevance between different modalities, fails to consider the multimodal content interaction within a document and between different documents, and cannot effectively utilize the information within multimodal documents, thus affecting the document retrieval effect and reducing the accuracy and relevance of document retrieval. Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0120] Next, referring to the accompanying drawings, a multimodal document retrieval device based on a cross-modal mutual attention mechanism proposed according to an embodiment of this application is described.

[0121] Figure 4 This is a schematic diagram of the structure of a multimodal document retrieval device based on a cross-modal mutual attention mechanism according to an embodiment of this application.

[0122] like Figure 4 As shown, the multimodal document retrieval device 10 based on the cross-modal mutual attention mechanism includes: a first modeling module 100, a second modeling module 200, a third modeling module 300, a fourth modeling module 400, and a retrieval module 500.

[0123] The first modeling module 100 is used to model the multimodal representation of the document, and obtains the first vectorized representation sequence of the text mode and the second vectorized representation sequence of the image mode of the document.

[0124] The second modeling module 200 is used to jointly model the text modality and the image modality based on the first vectorized representation sequence and the second vectorized representation sequence, so as to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image respectively, and to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and to model the multimodal enhanced vectorized representation of the document.

[0125] The third modeling module 300 is used to model the correlation between the same and different modalities of the document and the target document based on the mutual attention mechanism, so as to obtain the multimodal document representation perceived by the target document.

[0126] The fourth modeling module 400 is used to model the autocorrelation within document modalities, and to fuse the self-attention vectorized representation and multimodal enhancement vectorized representation of the document to obtain a unified multimodal enhancement representation of the document.

[0127] The retrieval module 500 is used to calculate the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation, and to sort the at least one candidate document based on the relevance score to retrieve the relevant document.

[0128] Optionally, in one embodiment of this application, the device 10 further includes: a first conversion module and a second conversion module.

[0129] The first conversion module is used to preprocess the text data of the document before modeling the multimodal representation of the document, filter out text that does not meet the preset conditions, and convert the filtered text into a text sequence to obtain the text data used for modeling.

[0130] The second conversion module is used to preprocess the image data of the document, converting the images into pixel sequences and normalizing the mean and variance of the images to obtain image data for modeling.

[0131] Optionally, in one embodiment of this application, the first modeling module 100 includes: an encoding unit, a generation unit, and an extraction unit.

[0132] The encoding unit is used to encode the word vector representation of the text in the document based on the word vector embedding matrix, so as to obtain the first vectorized representation sequence of the document.

[0133] The generation unit is used to generate image candidate boxes for images in a document, based on a pre-trained Mask-RCNN network.

[0134] The extraction unit is used to extract features of image candidate boxes based on the ResNet-50 network to obtain the second vectorized representation sequence of the document.

[0135] Optionally, in one embodiment of this application, the second modeling module 200 includes: a first modeling unit and a second modeling unit.

[0136] The first modeling unit is used to jointly model the text modality and the image modality based on VilBERT, so as to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image.

[0137] The second modeling unit is used to model the interaction between different modalities of a document based on a multimodal mutual attention mechanism, and to model the relationship between different modalities within a document based on a correlation matrix, so as to obtain the interaction between different modalities within a document and model the multimodal enhanced vectorized representation of the document.

[0138] Optionally, in one embodiment of this application, the third modeling module 300 includes: a first interaction unit and a second interaction unit.

[0139] The first interaction unit is used to interact with the same modal content of the document and the target document based on the mutual attention mechanism, model the correlation between the same modal of the document and the target document, and obtain the cross-document same modal document enhancement representation perceived by the target document.

[0140] The second interaction unit is used to interact with different modal content of the document and the target document based on the cross-modal mutual attention mechanism between documents, model the correlation between multiple documents and multiple modalities, and obtain the multimodal document representation perceived by the target document.

[0141] Optionally, in one embodiment of this application, the fourth modeling module 400 includes: a third modeling unit, a first fusion unit, and a second fusion unit.

[0142] The third modeling unit is used to model the autocorrelation within document modalities based on the multi-head self-attention mechanism, and obtain the self-attention vectorized representation of the document.

[0143] The first fusion unit is used to fuse the self-attention vectorized representation of the document and the vectorized representation of various multimodal enhancements of the document to obtain a unified multimodal enhancement representation of the document modality.

[0144] The second fusion unit is used for multimodal enhanced unified representation based on different document modalities, and the fusion is used to obtain multimodal enhanced unified representation.

[0145] Optionally, in one embodiment of this application, the retrieval module 500 includes a calculation unit and a sorting unit.

[0146] The computation unit is used to calculate the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation.

[0147] A sorting unit is used to sort at least one candidate document based on the relevance score of each candidate document to determine the relevant documents.

[0148] It should be noted that the foregoing explanation of the embodiment of the multimodal document retrieval method based on the cross-modal mutual attention mechanism also applies to the multimodal document retrieval device based on the cross-modal mutual attention mechanism of this embodiment, and will not be repeated here.

[0149] The multimodal document retrieval device based on a cross-modal mutual attention mechanism proposed in this application can obtain a unified representation of enhanced document multimodality by modeling a multimodal representation of a document and modeling a target document-aware multimodal representation based on a cross-modal mutual attention mechanism between documents. A relevance score is then calculated to retrieve matching documents, thereby fully utilizing the multimodal information of the document, enhancing the relevance between different modalities, and improving the matching degree of the document retrieval results, making the retrieval results more accurate and reliable. This solves the problems in related technologies where the modeling of each modality ignores the relevance between different modalities, fails to consider the multimodal content interaction within a document and between different documents, and cannot effectively utilize the information within multimodal documents, thus affecting the document retrieval effect and reducing the accuracy and relevance of document retrieval. Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

[0150] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0151] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0152] When the processor 502 executes the program, it implements the multimodal document retrieval method based on the cross-modal mutual attention mechanism provided in the above embodiments.

[0153] Furthermore, electronic devices also include:

[0154] Communication interface 503 is used for communication between memory 501 and processor 502.

[0155] The memory 501 is used to store computer programs that can run on the processor 502.

[0156] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0157] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0158] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0159] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0160] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal document retrieval method based on a cross-modal mutual attention mechanism as described above.

[0161] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0162] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0163] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0164] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0165] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0166] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0167] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0168] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A multimodal document retrieval method based on a cross-modal mutual attention mechanism, characterized in that, Includes the following steps: The multimodal representation of the modeling document is obtained by obtaining a first vectorized representation sequence of the text modality and a second vectorized representation sequence of the image modality of the document. Based on the first and second vectorized representation sequences, the text modality and the image modality are jointly modeled to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image, respectively. This is to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism and to model the multimodal enhanced vectorized representation of the document. Based on the mutual attention mechanism, the correlation between the same and different modalities of the document and the target document is modeled to obtain the multimodal document representation perceived by the target document; Model the autocorrelation within document modalities, fuse the self-attention vectorized representation of the document and the multimodal enhancement vectorized representation to obtain the unified multimodal enhancement representation of the document; Based on the multimodal enhanced unified representation, the relevance score between the target document and at least one candidate document is calculated, and the at least one candidate document is sorted based on the relevance score to retrieve the relevant document; The modeling of autocorrelation within document modalities, combined with the self-attention vectorized representation of the document and the vectorized representation of the multimodal enhancement, yields a unified multimodal enhancement representation of the document, including: Based on the multi-head self-attention mechanism, the autocorrelation within document modalities is modeled to obtain the self-attention vectorized representation of the document; By fusing the self-attention vectorized representation of the document with the vectorized representation of various multimodal enhancements of the document, a unified multimodal enhancement representation of the document modality is obtained; The multimodal enhanced unified representation is obtained by fusing different modalities of a document.

2. The method according to claim 1, characterized in that, Before modeling the multimodal representation of the document, the following is also included: The text data of the document is preprocessed to filter out text that does not meet the preset conditions, and the filtered text is converted into a text sequence to obtain text data for modeling. The image data of the document is preprocessed by converting the images into pixel sequences and normalizing the mean and variance of the images to obtain image data for modeling.

3. The method according to claim 1, characterized in that, The multimodal representation of the modeled document yields a first vectorized representation sequence for the text modality and a second vectorized representation sequence for the image modality, including: Based on the word vector embedding matrix, the word vector representation of the text in the document is encoded to obtain the first vectorized representation sequence of the document; Based on the pre-trained Mask-RCNN network, image candidate boxes are generated for the images in the document; Based on the ResNet-50 network, the features of the candidate bounding boxes of the image are extracted to obtain the second vectorized representation sequence of the document.

4. The method according to claim 1, characterized in that, The step involves jointly modeling the text modality and the image modality based on the first and second vectorized representation sequences to obtain multimodal vectorized representations of the document text and document images, respectively. This modeling, based on a multimodal mutual attention mechanism, models the interactions between different modalities of the document, thereby enhancing the multimodal vectorized representation of the document. Based on VilBERT, the text modality and the image modality are jointly modeled to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image; The interaction between different modalities of a document is modeled based on the multimodal mutual attention mechanism, and the relationship between different modalities within a document is modeled based on the relevance matrix to obtain the interaction between different modalities within the document, thereby modeling the multimodal enhanced vectorized representation of the document.

5. The method according to claim 1, characterized in that, The process of modeling the correlation between the same and different modalities of the document and the target document based on the mutual attention mechanism to obtain the multimodal document representation perceived by the target document includes: Based on the mutual attention mechanism, the same-modal content of the document and the target document are interacted, the correlation between the same modalities of the document and the target document is modeled, and the cross-document same-modal document augmentation representation perceived by the target document is obtained; Based on the cross-modal mutual attention mechanism between documents, the different modal contents of the document and the target document are interacted, the correlation between multiple documents and multiple modalities is modeled, and the multimodal document representation perceived by the target document is obtained.

6. The method according to claim 1, characterized in that, The process of calculating the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation, and ranking the at least one candidate document based on the relevance score to retrieve relevant documents includes: Based on the multimodal enhanced unified representation, the relevance score between the target document and the at least one candidate document is calculated; Based on the relevance score of each candidate document, the at least one candidate document is sorted to determine the relevant documents.

7. A multimodal document retrieval device based on a cross-modal mutual attention mechanism, characterized in that, include: The first modeling module is used to model the multimodal representation of the document, and obtains the first vectorized representation sequence of the text modality and the second vectorized representation sequence of the image modality of the document respectively; The second modeling module is used to jointly model the text modality and the image modality based on the first vectorized representation sequence and the second vectorized representation sequence, so as to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image respectively, so as to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and model the multimodal enhanced vectorized representation of the document. The third modeling module is used to model the correlation between the same and different modalities of the document and the target document based on the mutual attention mechanism, so as to obtain the multimodal document representation perceived by the target document; The fourth modeling module is used to model the autocorrelation within document modalities, and to fuse the self-attention vectorized representation of the document and the multimodal enhancement vectorized representation to obtain the unified multimodal enhancement representation of the document; The retrieval module is used to calculate the relevance score between the target document and at least one candidate document based on the multimodal enhanced unified representation, and to sort the at least one candidate document based on the relevance score to retrieve the relevant document; The fourth modeling module includes: The third modeling unit is used to model the autocorrelation within the document modality based on the multi-head self-attention mechanism, and obtain the self-attention vectorized representation of the document; The first fusion unit is used to fuse the self-attention vectorized representation of the document and the vectorized representation of various multimodal enhancements of the document to obtain a unified multimodal enhancement representation of the document modality; The second fusion unit is used to fuse the multimodal enhanced unified representation based on different modalities of the document to obtain the multimodal enhanced unified representation.

8. The apparatus according to claim 7, characterized in that, Also includes: The first conversion module is used to preprocess the text data of the document before modeling the multimodal representation of the document, filter out text that does not meet the preset conditions, and convert the filtered text into a text sequence to obtain text data for modeling. The second conversion module is used to preprocess the image data of the document, convert the images into pixel sequences, and normalize the mean and variance of the images to obtain image data for modeling.

9. The apparatus according to claim 7, characterized in that, The first modeling module includes: The encoding unit is used to encode the word vector representation of the text in the document based on the word vector embedding matrix, so as to obtain the first vectorized representation sequence of the document; A generation unit is used to generate image candidate boxes for images in the document based on a pre-trained Mask-RCNN network. An extraction unit is used to extract features of the image candidate boxes based on the ResNet-50 network to obtain a second vectorized representation sequence of the document.

10. The apparatus according to claim 7, characterized in that, The second modeling module includes: The first modeling unit is used to jointly model the text modality and the image modality based on VilBERT to obtain the multimodal vectorized representation of the text and the multimodal vectorized representation of the image; The second modeling unit is used to model the interaction between different modalities of the document based on the multimodal mutual attention mechanism, and to model the mutual relationship between different modalities within the document based on the relevance matrix, so as to obtain the interaction between different modalities within the document and model the multimodal enhanced vectorized representation of the document.

11. The apparatus according to claim 7, characterized in that, The third modeling module includes: The first interaction unit is used to interact with the same modal content of the document and the target document based on the mutual attention mechanism, model the correlation between the same modal content of the document and the target document, and obtain the cross-document same modal document augmentation representation perceived by the target document; The second interaction unit is used to interact with the different modal content of the document and the target document based on the cross-modal mutual attention mechanism between documents, model the correlation between multiple documents and multiple modalities, and obtain the multimodal document representation perceived by the target document.

12. The apparatus according to claim 7, characterized in that, The retrieval module includes: The calculation unit is used to calculate the relevance score between the target document and the at least one candidate document based on the multimodal enhanced unified representation; A sorting unit is used to sort the at least one candidate document based on the relevance score of each candidate document to determine the relevant documents.

13. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal document retrieval method based on a cross-modal mutual attention mechanism as described in any one of claims 1-6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multimodal document retrieval method based on cross-modal mutual attention mechanism as described in any one of claims 1-6.

Images