Document-oriented standard element and application scene semantic integration multi-modal retrieval method

Abstract

The invention provides a document-oriented standard element and application scene semantic integration multi-modal retrieval method, and belongs to the technical field of information retrieval. Comprising the steps of obtaining different modal data of a source document, performing feature extraction on the different modal data of the source document in all directions, and fusing extracted source document features into a unified semantic representation vector; extracting feature information of each direction of the target document, and fusing the feature information into a unified semantic representation vector of the target document; generating a plurality of matching results to be selected according to the multi-scale similarity of the source document and the target document through a multi-modal alignment algorithm; a final matching result is obtained by selecting the to-be-selected matching result with the maximum total similarity; the method aims at achieving collaborative retrieval of texts, images and structured metadata through a semantic modeling technology. In the similarity calculation process, the total similarity of the source document and the target document is fully considered, the similarity in the single direction is kept at a certain level, and a matching result better conforms to a real result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, in particular to a multi-modal retrieval method for integrating standard elements of a document with semantics of an application scenario. BACKGROUND

[0002] In the era of information explosion, with the continuous increase of various document contents, the traditional text retrieval method has been difficult to meet the needs of users in complex application scenarios; the existing document retrieval technology mostly depends on keyword matching or simple semantic analysis, which leads to low relevance and accuracy of the retrieval results, especially in the application scenarios where multi-modal data (such as text, image, audio and video, etc.) coexist, a single text retrieval method cannot effectively integrate multiple information sources; therefore, how to effectively integrate the standard elements in the document with the semantic information of the application scenario, and use multi-modal technology to improve the accuracy and efficiency of retrieval, has become a research hotspot; the existing retrieval methods mainly have the following problems: single-modal retrieval bottleneck, shallow feature fusion defect and language gap problem; and the traditional multi-modal retrieval method mostly depends on independent modal processing, and also lacks a unified semantic model, which leads to poor integration of information between different modalities, affecting the retrieval effect; and when calculating the similarity between the source document and the target document, mainly focusing on the size of the total similarity, ignoring the influence of low similarity in a single direction on the matching result, which will lead to the phenomenon that the matching result is inaccurate in some dimensions. SUMMARY

[0003] The purpose of the present application is to overcome the above problems existing in the prior art and greatly improve the technical effect based on the original technology; the present application provides a multi-modal retrieval method for integrating standard elements of a document with semantics of an application scenario, which comprises:

[0004] S1, collecting first modal source document data of standard elements and application scenarios through a source document acquisition module, the source document acquisition module refers to a module responsible for acquiring, collecting, storing and managing source documents, and the source document data includes: text data, associated image data and structured metadata; the source document acquisition module refers to a module responsible for acquiring, collecting, storing and managing source documents.

[0005] S2, processing the source document data of the first modality using a multi-modal semantic encoding module, the processing comprising: S21, performing domain term recognition and word segmentation processing on the text data of the source document data of the first modality of the standard element and the application scenario; and encoding the text data processed by word segmentation through a text encoder to generate a text word granularity semantic encoding feature vector sequence; S22, processing the associated image data in the source document data of the first modality through a convolutional neural network to extract a visual feature vector sequence of the associated image; S23, processing the structured metadata in the source document data of the first modality through a graph neural network to generate a metadata embedding vector; the multi-modal semantic encoding module refers to a part that converts data of different modalities into a unified semantic space; the data of different modalities include: text data, image data and structured metadata; the text encoder is a pre-trained language model based on a Transformer architecture, which models the text sequence through a self-attention mechanism to generate a text word granularity semantic encoding feature vector sequence; the convolutional neural network extracts associated image features through convolutional layers and pooling layers to generate a visual feature vector sequence; the graph neural network processes the structured metadata using graph convolution operations to generate an embedding vector of the structured metadata, the embedding vector including semantic information and adjacency relationships of nodes; the nodes refer to basic elements in the text data and the associated image, including: users, entities and words.

[0006] S3, realizing multi-modal semantic integration through a standard element fusion module; the method for realizing multi-modal semantic integration comprises: establishing a cross-attention matrix of source document text features, visual features and metadata features through the vectors generated by S2; generating a unified semantic representation vector of the source document using a gated fusion mechanism, and dynamically adjusting the gating weight according to the standard type; the standard element fusion module refers to the fusion of the feature vectors of the text data, the associated image data and the structured metadata to generate a unified and more expressive feature representation; the vectors generated by S2 are fused through a cross-attention mechanism based on a Transformer model to establish a cross-attention matrix of source document text features, visual features and metadata features; the vectors generated by S2 include: a text word granularity semantic encoding feature vector sequence, a visual feature vector sequence of the associated image and a metadata embedding vector; the gated fusion mechanism dynamically adjusts the gating weight according to the standard type of the source document to control the fusion proportion of the text features, the visual features and the metadata features; the standard type of the source document refers to the standardized type of the format, structure and representation of the source document.

[0007] S4, obtaining the second modality target document of the standard element and the application scene by a target document processing module, and generating the target document semantic representation vector using the multi-modal semantic encoding module in S2; the target document processing module refers to a module responsible for obtaining, collecting, storing and managing target document data; the second modality target document includes: the data, associated image data and structured metadata of the target document; the target document semantic representation vector includes: the text word granularity semantic encoding feature vector sequence, the visual feature vector sequence and the metadata embedding vector of the target document.

[0008] S5, calculating the multi-scale similarity of the source document unified semantic representation vector and the target document semantic representation vector by a multi-modal alignment module, respectively judging whether each scale similarity exceeds the corresponding threshold range to generate a plurality of selected matching results; calculating the total similarity of the plurality of selected matching results respectively, and taking the selected matching result with the maximum total similarity as the final matching result; the multi-modal alignment module realizes the multi-scale similarity calculation of the representation vector through the existing unified semantic space representation vector, multi-scale feature extraction, similarity measurement and fusion mechanism; the existing unified semantic space representation vector refers to the unified semantic representation vector generated by the gating fusion mechanism; the respective judgment of whether each scale similarity exceeds the corresponding threshold range includes: respectively judging whether the similarity of each scale in each template data of the source document is greater than or equal to 0.7, if the similarity of all scales is greater than or equal to 0.7, the corresponding template data is taken as a selected matching result, until a plurality of selected matching results are obtained; each template data refers to a complete document data in the source document data; the respective calculation of the total similarity of the plurality of selected matching results includes: using the weighted summation method to add up the respective scale similarities of each selected matching result to calculate the total similarity, and taking the selected matching result with the maximum total similarity as the final matching result.

[0009] The beneficial effects of the present application are:

[0010] The present application provides a document-oriented standard element and application scene semantic integrated multi-modal retrieval method; has the following advantages:

[0011] 1. The present application provides a document-oriented standard element and application scene semantic integrated multi-modal retrieval method, which aims to effectively realize the collaborative retrieval of various modal data such as text, image and structured metadata by combining deep learning and semantic modeling technology, improve the semantic relevance and precision of the retrieval result, and solve the defect that single modal retrieval cannot effectively fuse multiple information sources.

[0012] 2, in the similarity calculation of the source document and the target document, firstly, when judging the similarity of each scale in each template data of the source document, the corresponding template data with the similarity of all scales greater than a certain value is taken as the selected matching result, and multiple selected matching results are obtained; then, the total similarity of the multiple selected matching results is calculated, and the selected matching result with the highest total similarity is taken as the final matching result; therefore, the final matching result not only fully considers the total similarity of the source document and the target document, but also keeps the one-way similarity at a certain level, and the matching result will be more consistent with the real result. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The flow chart of the document-oriented standard element and application scene semantic integrated multi-modal retrieval method of the application. DETAILED DESCRIPTION

[0014] The specific embodiments of the application will be described in detail below with reference to the accompanying drawings; it should be understood that the specific embodiments given herein are only for the purpose of illustrating and explaining the application, and cannot be used to limit the application.

[0015] It should be noted that in the following description, many specific details are set forth in order to provide a thorough understanding of the application, however, the application can also have other implementations and variations, therefore, the protection scope of the application is not limited by the specific embodiments disclosed below.

[0016] As Figure 1As shown, the flowchart of the multi-modal retrieval method integrating standard elements and application scene semantics according to the embodiment of the application; the flowchart comprises: step S1, collecting first modal source document data of standard elements and application scene through a source document collection module, the source document collection module refers to a module responsible for obtaining, collecting, storing and managing source documents, and the source document data comprises: text data, associated image data and structured metadata; step S2, processing the first modal source document data using a multi-modal semantic encoding module, the processing steps are: S21, performing field term recognition and word segmentation processing on the text data of the first modal source document data of standard elements and application scene; and encoding the text data processed by word segmentation using a text encoder to generate a text word granularity semantic encoding feature vector sequence; S22, processing the associated image data in the first modal source document data through a convolutional neural network to extract a visual feature vector sequence of the associated image; S23, processing the structured metadata in the first modal source document data through a graph neural network to generate a metadata embedding vector; step S3, realizing multi-modal semantic integration through a standard element fusion module; the method of realizing multi-modal semantic integration is: establishing a cross-attention matrix of source document text features, visual features and metadata features through the vectors generated in S2; generating a unified semantic representation vector of the source document using a gated fusion mechanism, and dynamically adjusting the gate weight according to the standard type; step S4, obtaining a second modal target document of standard elements and application scene through a target document processing module, and generating a target document semantic representation vector using the multi-modal semantic encoding module in S2; step S5, calculating the multi-scale similarity of the source document unified semantic representation vector and the target document semantic representation vector through a multi-modal alignment module, respectively determining whether each scale similarity exceeds the corresponding threshold range, and generating a plurality of selected matching results; respectively calculating the total similarity of the plurality of selected matching results, and taking the selected matching result with the maximum total similarity as the final matching result.

[0017] In step S1, the source document collection module refers to a module responsible for obtaining, collecting, storing and managing source documents; that is, the source document data refers to data information in historical documents, by collecting data information in historical documents, including: text information, image information and structured metadata information, to facilitate the extraction of features of each modal data information in the later stage.

[0018] In step S2, the multi-modal semantic encoding module refers to the part that converts data of different modalities into a unified semantic space representation, for converting data of different modalities into a unified form representation; the data of different modalities include: text data, image data and structured metadata; the text encoder is a pre-trained language model based on the Transformer architecture, which models the text sequence through a self-attention mechanism to generate a text word granularity semantic encoding feature vector sequence; the convolutional neural network extracts associated image features through convolutional layers and pooling layers to generate a visual feature vector sequence; the graph neural network processes the structured metadata using graph convolution operations to generate an embedding vector of the structured metadata, which includes semantic information and adjacency relationships of nodes; the nodes refer to basic elements in the text data and associated images, including: users, entities and words; therefore, the structured metadata information includes semantic information and relationship information of users, entities and words.

[0019] In step S3, the standard element fusion module refers to the module that fuses the feature vectors of text data, associated image data and structured metadata to generate a unified and more expressive feature representation; the vectors generated in S2 are fused through a cross-attention mechanism based on the Transformer model to establish a cross-attention matrix of the text features, visual features and metadata features of the source document; the vectors generated in S2 include: text word granularity semantic encoding feature vector sequence, associated image visual feature vector sequence and metadata embedding vector.

[0020] Specifically, the gating fusion mechanism dynamically adjusts the gating weights according to the standard type of the source document to control the fusion ratio of the text features, visual features and metadata features; the standard type of the source document refers to the standardized type of the format, structure and representation of the source document.

[0021] In step S4, the target document processing module refers to the module responsible for acquiring, collecting, storing and managing target document data; the second modal target document includes: text data, associated image data and structured metadata of the target document; the target document semantic representation vector includes: text word granularity semantic encoding feature vector sequence, visual feature vector sequence and metadata embedding vector of the target document.

[0022] In step S5, the multi-modal alignment module realizes multi-scale similarity calculation of the representation vectors by including the existing unified semantic space representation vector, multi-scale feature extraction, similarity measurement and fusion mechanism; the existing unified semantic space representation vector refers to a unified semantic representation vector generated by a gating fusion mechanism; the respective judgment whether each scale similarity exceeds a corresponding threshold range includes: respectively judging whether the similarity of each scale in each template data of the source document is greater than or equal to 0.7, if the similarity of all scales is greater than or equal to 0.7, the corresponding template data is taken as a selected matching result, until a plurality of selected matching results are obtained; the respective template data refers to a complete document data in the source document data; the respective calculation of the total similarity of the plurality of selected matching results includes: using a weighted summation method to add up the respective scale similarities of each selected matching result to calculate the total similarity, and taking the selected matching result with the maximum total similarity as the final matching result.

[0023] Specifically, the similarity calculation method of each scale adopts cosine similarity for calculation; assuming that the semantic representation of the source document in one scale is V s , the semantic representation of the target document in one scale is V t , and the calculation formula of the cosine similarity S s of the source document and the target document in the scale is:

[0024] wherein, S s is the cosine similarity in the corresponding scale, and ||·|| represents the norm of the vector.

[0025] When the weighted summation is performed, because the similarity in each scale of the target document and the document of the selected matching result is greater than or equal to 0.7, the value of the total similarity is also greater than or equal to 0.7; assuming that S s1 is the similarity of the target document compared with the document of a selected matching result in the text information scale, S s2 is the similarity of the target document compared with the document of a selected matching result in the associated image information scale, and S s3 is the similarity of the target document compared with the document of a selected matching result in the structured metadata information scale, and α1, α2 and α3 are respectively the weights of S s1 , S s2 and S s3 , then α1+α2+α3=1, if the importance of the matching result is required to be: structured metadata>text data>associated image, then α3>α1>α2; and the calculation formula of the total similarity is:

[0026] S 总 =α1·S s1 +α2·S s2 +α3·Ss3

[0027] The total similarity of all the candidate matching results is calculated, and the source document of the candidate matching result with the maximum total similarity is taken as the final matching result.

Claims

1. A document-oriented standard element and application scenario semantic integrated multi-modal retrieval method, characterized in that, The method comprises: S1, collecting first modal source document data of standard elements and application scenarios by a source document collection module, the source document collection module being a module responsible for obtaining, collecting, storing and managing source documents, and the source document data comprising text data, associated image data and structured metadata; S2, processing the source document data of the first modal using a multi-modal semantic encoding module, the processing steps being: S21, performing domain term recognition and word segmentation processing on the text data of the source document data of the first modal of standard elements and application scenarios; and encoding the text data processed by word segmentation using a text encoder to generate a text word granularity semantic encoding feature vector sequence; S22, processing the associated image data in the source document data of the first modal using a convolutional neural network to extract a visual feature vector sequence of the associated image; S23, processing the structured metadata in the source document data of the first modal using a graph neural network to generate a metadata embedding vector; S3, realizing multi-modal semantic integration by a standard element fusion module; the method for realizing multi-modal semantic integration being: establishing a cross-attention matrix of source document text features, visual features and metadata features using the vectors generated in S2; generating a unified semantic representation vector of the source document using a gated fusion mechanism, and dynamically adjusting the gating weight according to the standard type; S4, obtaining a second modal target document of standard elements and application scenarios by a target document processing module, and generating a target document semantic representation vector using the multi-modal semantic encoding module in S2; S5, calculating multi-scale similarity of the unified semantic representation vector of the source document and the semantic representation vector of the target document by a multi-modal alignment module, respectively judging whether each scale similarity exceeds the corresponding threshold range to generate a plurality of selected matching results; and respectively calculating the total similarity of the plurality of selected matching results, and taking the selected matching result with the maximum total similarity as the final matching result. 2.The document-oriented standard element and application scenario semantic integrated multi-modal retrieval method according to claim 1, characterized in that, The source document collection module is a module responsible for obtaining, collecting, storing and managing source documents. 3.The document-oriented standard element and application scenario semantic integrated multi-modal retrieval method of claim 1, wherein, The multi-modal semantic encoding module is a part that converts data of different modalities into a unified semantic space; the data of different modalities comprises text data, image data and structured metadata; the text encoder is a pre-trained language model based on a Transformer architecture, which models text sequences through a self-attention mechanism to generate a text word granularity semantic encoding feature vector sequence; the convolutional neural network extracts associated image features through convolutional layers and pooling layers to generate a visual feature vector sequence; the graph neural network processes structured metadata using graph convolution operations to generate an embedding vector of the structured metadata, the embedding vector comprising semantic information and adjacency relationships of nodes; the nodes refer to basic elements in the text data and the associated image, including users, entities and words.

4. The document-oriented standard element and application scenario semantic integrated multi-modal retrieval method according to claim 1, characterized in that, The standard element fusion module refers to fusing the feature vectors of text data, associated image data and structured metadata; the vectors generated by S2 are fused through a cross-attention mechanism based on a Transformer model to establish a cross-attention matrix of the text features, visual features and metadata features of the source document; the vectors generated by S2 include a text word granularity semantic encoding feature vector sequence, a visual feature vector sequence of associated images and a metadata embedding vector.

5. The document-oriented standard element and application scenario semantic integrated multi-modal retrieval method according to claim 1, characterized in that, The gating fusion mechanism dynamically adjusts the gating weight according to the standard type of the source document to control the fusion ratio of the text features, visual features and metadata features; the standard type of the source document refers to the standardized type of the format, structure and representation of the source document.

6. The document-oriented standard element and application scenario semantic integrated multi-modal retrieval method according to claim 1, characterized in that, The target document processing module refers to a module responsible for acquiring, collecting, storing and managing target document data; The second modality target document includes text data, associated image data and structured metadata of the target document; the target document semantic representation vector includes a text word granularity semantic encoding feature vector sequence, a visual feature vector sequence and a metadata embedding vector of the target document.

7. The document-oriented standard element and application scenario semantic integrated multi-modal retrieval method according to claim 1, characterized in that, The multi-modal alignment module includes an existing unified semantic space representation vector, multi-scale feature extraction, similarity measurement and a fusion mechanism to realize multi-scale similarity calculation of the representation vector; the existing unified semantic space representation vector refers to a unified semantic representation vector generated by the gating fusion mechanism; the step of respectively judging whether each scale similarity exceeds the corresponding threshold range includes respectively judging whether each scale similarity in each template data of the source document is greater than or equal to 0.7, if all scale similarities are greater than or equal to 0.7, the corresponding template data is taken as a selected matching result, until multiple selected matching results are obtained; each template data refers to a complete document data in the source document data; the step of respectively calculating the total similarity of the multiple selected matching results includes adding up each scale similarity of each selected matching result to calculate the total similarity by using a weighted summation method, and the selected matching result with the maximum total similarity is taken as the final matching result.

Images