A method, device and medium for multi-modal feature alignment of heterogeneous data

Abstract

The application discloses a multi-modal feature alignment method for heterogeneous data, equipment and medium, and relates to the field of intelligent retrieval. The method comprises the following steps: analyzing an input multi-modal document to identify text elements and non-text elements; extracting text original features, image original features and structured text original features; constructing a shared semantic space; mapping the image original features to the shared semantic space through an image mapping network, and mapping the text original features and the structured text original features to the shared semantic space through a text mapping network; based on the mapped features in the shared semantic space, performing multi-objective joint optimization through a pre-defined multi-objective optimization function to drive feature alignment; and generating a query vector based on the aligned image alignment features and the text alignment features. By constructing a unified shared semantic space and utilizing multi-objective joint optimization to drive feature alignment, the original information of the non-text elements can be completely preserved in the feature mapping process.

Description

Technical Field

[0001] This application relates to the field of intelligent retrieval technology, specifically to a method, device, and medium for aligning multimodal features of heterogeneous data. Background Technology

[0002] With the widespread application of intelligent retrieval technology in enterprise knowledge bases, academic search engines, and professional document management systems, the demand for accurate retrieval of heterogeneous content such as text, images, and tables is becoming increasingly prominent. In professional scenarios such as technical manuals, research papers, and financial reports, information is usually composed of heterogeneous data consisting of text and non-text elements such as images, flowcharts, and data tables. Achieving unified and accurate understanding and retrieval of heterogeneous data has become key to improving the level of industry intelligence.

[0003] Existing methods for processing heterogeneous data typically employ a phased process. First, document parsing techniques are used to identify and separate text and image elements. Then, a visual language model is used to convert the image into a natural language descriptive text. Finally, the original text and the generated image descriptive text are converted into vectors using a unified text encoder, and semantic retrieval is performed within this vector space using metrics such as cosine similarity. This approach to processing heterogeneous data compresses the original data during the conversion between different modalities, resulting in the significant loss of spatial structure, detailed features, and domain-specific symbols. Furthermore, because images and text have fundamentally different distribution characteristics in the original feature space, forcibly using a vector space optimized for text and similarity metrics for cross-modal retrieval leads to inefficient semantic matching of image features, resulting in low accuracy in image content retrieval. Summary of the Invention

[0004] To address the aforementioned issues, this application proposes a multimodal feature alignment method for heterogeneous data, comprising: The input multimodal document is parsed to identify text elements and non-text elements; the non-text elements include natural image data and graphical data with logical structure. The text feature extraction engine extracts the original text features of the text elements, and based on the information carrying type of the non-text elements, the corresponding non-text feature extraction engine is called to extract the original image features of the natural image data and the original structured text features of the logically structured graphic data, respectively. A shared semantic space is constructed, and the original features of the image are mapped to the shared semantic space through an image mapping network, and the original features of the text and the original features of the structured text are mapped to the shared semantic space through a text mapping network. Based on the mapping features in the shared semantic space, multi-objective joint optimization is performed through a predefined multi-objective optimization function to drive feature alignment; Based on the aligned image alignment features and text alignment features, a query vector is generated to perform a query in the index.

[0005] On the other hand, this application also proposes a multimodal feature alignment device for heterogeneous data, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform, for example, a multimodal feature alignment method for heterogeneous data as described in the above example.

[0006] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as: a multimodal feature alignment method for heterogeneous data as described in the above example.

[0007] The proposed method for aligning multimodal features of heterogeneous data in this application offers the following advantages: By constructing a unified shared semantic space and using multi-objective joint optimization to drive feature alignment, the semantic matching accuracy of image and text features is significantly improved, ensuring that the original information of non-text elements is fully preserved during the feature mapping process, thereby achieving more accurate and relevant cross-modal retrieval results in intelligent retrieval applications.

[0008] Furthermore, the feature alignment mechanism reduces the reliance on large-scale pre-trained models and a large amount of labeled data, lowers the consumption of computing resources, and improves the practicality and scalability of the system. It can be flexibly applied to various business scenarios, such as technical document retrieval and multimodal question answering systems, providing reliable technical support for the efficient processing and intelligent application of heterogeneous data. Attached Figure Description

[0009] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a method for aligning multimodal features of heterogeneous data according to an embodiment of this application. Figure 2 This is a schematic diagram of a multimodal feature alignment device for heterogeneous data in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0012] like Figure 1 As shown in the figure, this application provides a method for aligning multimodal features of heterogeneous data, including: S101. Parse the input multimodal document to identify text elements and non-text elements; the non-text elements include natural image data and graphic data with logical structure.

[0013] A pre-trained layout analysis model is used to perform semantic region detection on input multimodal documents. This model, built on a deep convolutional neural network, is capable of identifying different semantic regions within the document and obtaining multiple candidate regions. Specifically, the layout analysis model receives an image representation of the document, obtains feature representations of each region through a feature extraction network, and then generates candidate region bounding boxes through a region proposal network.

[0014] Furthermore, a multi-feature fusion analysis is performed on each candidate region. Specifically, visual and texture features are extracted for each candidate region. Visual features include color distribution, shape features, and edge information, while texture features are extracted using a local binary pattern algorithm. Based on the visual and texture features, a pre-trained classifier is used to classify the candidate regions into text regions and non-text regions.

[0015] It should be noted that the classifier uses the support vector machine algorithm and is trained on a dataset containing multiple document types.

[0016] For the non-text regions identified through classification, further fine-grained analysis is conducted to identify the specific information carrying type. The geometric features, structural complexity, and content composition of the non-text regions are analyzed, distinguishing them into natural image data and graphical data with logical structures. Graphical data with logical structures includes at least two types: flowcharts and data tables. Flowchart identification is based on its connecting lines, arrow symbols, and geometric shape combinations; data table identification is based on its regular row and column structure and dividing lines.

[0017] It's important to note that images and text differ fundamentally in how they carry information. For example, a standard 512×512 pixel color image can carry approximately 780,000 dimensions of original information, while a text description containing the same semantics typically contains only 50-200 words. This vast difference in information density is the root cause of information loss during the conversion process. From an information theory perspective, let image information be... Its information entropy is: The corresponding text sequence The conditional information entropy is: .

[0018] The essential difference between the two at the information theory level lies in two aspects: parallelism and sequentiality. Specifically, images have spatial parallelism, and a single pixel matrix can simultaneously encode multi-level information such as color, texture, and shape; while text is a sequential linear information flow, which must express semantics through the sequential combination of words.

[0019] Meanwhile, the image-to-text conversion process is an extremely lossy compression process, but text descriptions often only extract coarse-grained semantics. Information lost during the conversion, such as spatial relationships, color coding, and symbolic details, is crucial for retrieval in many fields. This difference in information density means that the specialized knowledge clearly expressed in the image becomes blurred and difficult to retrieve after being converted to text descriptions.

[0020] Image features and text features exhibit drastically different distribution characteristics in high-dimensional vector spaces. In a d-dimensional vector space... In this context, the text vectors generated by the text encoder approximately follow a Gaussian distribution: This distribution is relatively regular, and the semantic differences between different texts are mainly reflected in the changes in vector direction. The pixel distribution of natural images is constrained by the physical world's geometric structure, lighting conditions, and viewpoint changes, resulting in a complex nonlinear structure in the feature space. Therefore, the image vectors generated by the visual encoder exhibit a nonlinear manifold structure. , where M is a low-dimensional nonlinear manifold embedded in a high-dimensional space.

[0021] When using a uniform distance metric such as cosine similarity to perform retrieval in the same vector space, it is essentially using textual metrics to measure image features, which can easily lead to a significant deterioration in image retrieval performance.

[0022] S102. Using a text feature extraction engine, extract the original text features of the text elements, and based on the information carrying type of the non-text elements, call the corresponding non-text feature extraction engine to extract the original image features of the natural image data and the original structured text features of the graphic data containing logical structures.

[0023] For text elements, the identified text elements are input into the text feature extraction engine. The text feature extraction engine is built on a text encoder, which preprocesses the text, including character encoding unification, special symbol filtering, and text length standardization. The text encoder performs multi-level semantic parsing of the input text through a 12-layer Transformer structure, sequentially performing word embedding mapping, positional encoding overlay, and multi-head self-attention calculation, finally outputting a 768-dimensional original text feature vector.

[0024] For non-text elements, the visual encoder is invoked for the identified natural image data. The visual encoder is a visual Transformer architecture based on the multimodal large model (BLIP-2 model). It adjusts the input image to a preset pixel resolution and performs normalization processing. The image is segmented into several image patches and flattened into a sequence through the patch embedding layer of the encoder. The feature transformation is performed through 12 visual Transformer layers. Each layer performs layer normalization, multi-head self-attention and feedforward network calculation in sequence. The feature representation corresponding to the label is extracted from the last layer as the original image feature.

[0025] For flowcharts, data tables, and other graphical data with logical structures, during the structure analysis stage, the contour detection algorithm in the cross-platform computer vision library (OpenCV library) is used to identify the basic geometric elements in the graphics. Optical character recognition (OCR) is performed on the detected text regions to extract the text content. Based on spatial positional relationships and connection line analysis, logical connections between elements are established.

[0026] During the structured description generation phase, for flowcharts, text descriptions are generated based on preset templates; for data tables, table content is organized based on preset formats, such as row number-column name:value format. Structural semantic tags are added to the generated structured text content, such as " <row> ”、" <cell>XML tags such as "".

[0027] The generated structured text description is input into the text feature extraction engine. Following the same processing flow as plain text elements, features are extracted through the text encoder, and the output is a structured text original feature vector with the same feature dimensions as the original text.

[0028] By managing different feature extraction engines through a unified interface, it ensures that various features meet the input requirements of subsequent mapping networks in terms of dimension, accuracy, and numerical range.

[0029] S103. Construct a shared semantic space, map the original features of the image to the shared semantic space through an image mapping network, and map the original features of the text and the original features of the structured text to the shared semantic space through a text mapping network.

[0030] Specifically, the dimensional parameters of the shared semantic space are predefined. It should be noted that the predefined dimensional parameters are uniformly applicable to feature representations of all modalities. A spatial metric is established, using cosine similarity as the similarity criterion for feature vectors within the space. Feature standardization constraints are set, requiring all feature vectors mapped to the shared semantic space to be L2 norm normalized to complete the initialization of the shared semantic space.

[0031] Furthermore, based on the spatial dimension parameter, the weight parameters of the image mapping network and the text mapping network are initialized. The weight parameters of the image mapping network and the text mapping network adopt the same network structure but the parameters are completely independent.

[0032] The image mapping network employs a multi-layer feedforward neural network structure. The first layer performs a linear transformation, receiving the original image feature vector and adjusting its dimension using a linear transformation matrix, converting the input features from the original dimension to the dimension of the intermediate hidden layer. The second layer performs non-linear activation, applying a non-linear activation function to the linearly transformed features, introducing the model's non-linear expressive power. The third layer performs dimension compression, performing a second linear transformation on the activated features to compress the feature dimension to a specified dimension within the target shared semantic space. After the transformation is complete, standardization is performed, calculating the L2 norm of the output features and dividing the feature vector by its norm value to achieve unit vectorization.

[0033] The text mapping network has the same hierarchical structure as the image mapping network, but its parameters are independent. The text mapping network uniformly receives the original text features and the original structured text features as input, and performs the same three-layer transformation operation as the image mapping network, including linear transformation, nonlinear activation, and quadratic linear transformation. In the output stage, L2 norm normalization is also performed to ensure that all text-related features are mapped to the same semantic space coordinate system as the image features.

[0034] In this embodiment, the image mapping network and text mapping network employ a three-layer MLP structure to learn the complex nonlinear transformation from the original feature space to the shared semantic space. Each layer plays a different functional role. The image feature mapping network structure is as follows: .

[0035] Among them, the first layer A dimensionality transformation is performed to extend the 768-dimensional BLIP-2 features to a higher dimension, providing sufficient expressive power to capture visual semantic details; the second layer... Learning nonlinear semantic transformations, this layer is the core semantic layer, achieved through nonlinear activation functions. (Using GELU) to model complex cross-modal semantic relationships; third layer It performs dimensionality compression and fine-grained adjustment, projects features onto the target shared space, and also serves as a regularization function.

[0036] The text feature mapping network structure is as follows: They use the same three-layer structure but with completely independent parameters. Because text features and image features have different distributions, forcing shared parameters weakens the model's ability to learn the characteristics of their respective modalities. Independent parameters allow each mapper to specifically optimize the feature transformation for its own modality.

[0037] Even after mapping, the vector magnitude distributions of features from different modalities still differ, and the problems of eigenvector magnitude explosion or vanishing are prone to occur. Therefore, the mapped features need to be L2 standardized and projected onto a unit hypersphere, as shown in the formula: ,in The standardized feature vectors are compared and learned on a unit sphere, making gradient updates more stable. Standardization ensures that the comparison is of direction rather than magnitude. At this point, cosine similarity is equivalent to Euclidean distance on a unit sphere, simplifying subsequent retrieval calculations.

[0038] S104. Based on the mapping features in the shared semantic space, perform multi-objective joint optimization through a predefined multi-objective optimization function to drive feature alignment.

[0039] A multi-objective optimization function is constructed. First, the similarity matrix between all image mapping features and text mapping features in the shared semantic space is calculated. Each element s_ij in the similarity matrix represents the inner product similarity between the i-th image mapping feature and the j-th text mapping feature. Based on the similarity matrix, a first loss function is constructed. It should be noted that the first loss function uses normalized temperature-scaled cross-entropy to promote positive sample pairs to be closer together and negative sample pairs to be further apart.

[0040] Secondly, the difference in mean embeddings between the set of image mapping features and the set of text mapping features in the regenerating kernel Hilbert space is calculated. Specifically, this involves mapping all features to the regenerating kernel Hilbert space using a radial basis function, calculating the mean embeddings of the image mapping features and text mapping features in this space respectively, then calculating the squared Euclidean distance between the two mean embeddings, and constructing a second loss function based on this squared Euclidean distance.

[0041] Finally, the k-nearest neighbor distributions of corresponding features are calculated in both the original image feature space and the shared semantic space. For each image mapping feature, its k nearest neighbors are determined in the original feature space, and the k nearest neighbors of the same feature are determined in the shared semantic space. The divergence between the two nearest neighbor distributions is then calculated. Text features are processed similarly, and a third loss function is constructed based on the divergence values.

[0042] The first loss function, the second loss function, and the third loss function are combined according to preset weights to construct a multi-objective optimization function.

[0043] Based on the constructed multi-objective optimization function, joint optimization of objectives is performed on the image mapping network and the text mapping network to drive feature alignment of the mapping.

[0044] Specifically, in each training iteration, a batch containing N pairs of image-text samples is sampled from the training set, and the samples are passed through the corresponding mapping network to obtain the mapping features in the shared semantic space.

[0045] Calculate the multi-objective optimization function value for the current batch, calculate the feature distribution loss value corresponding to the first loss function based on the mapping features to narrow the feature distance between related image and text pairs; calculate the information fidelity loss value corresponding to the second loss function based on the mapping features to reduce the overall difference between image and text feature distributions; calculate the semantic similarity loss value corresponding to the third loss function based on the mapping features to maintain the semantic consistency of features before and after mapping.

[0046] Furthermore, the backpropagation algorithm is executed based on the total loss value to calculate the partial derivatives of the total loss with respect to the parameters of the image mapping network and the text mapping network, thereby obtaining the parameter gradients. Based on the gradient descent principle and the preset learning rate, the update amount of each parameter is determined.

[0047] Simultaneously update the parameters of both mapping networks. Based on the calculated update amounts, adjust the weight matrix and bias vector in the image mapping network, and synchronously adjust the weight matrix and bias vector in the text mapping network. Repeat the parameter optimization process until the change in the total loss value is less than a preset threshold or the maximum number of training epochs is reached, thus completing the feature alignment process.

[0048] In this embodiment, an optimization function is constructed that includes cross-modal semantic similarity loss, feature distribution consistency loss, and information fidelity loss. This function simultaneously constrains the learning process of the mapping network from three dimensions: semantic similarity, distribution consistency, and information fidelity, ensuring the integrity of the original information of non-text elements and the rationality of feature distribution during feature alignment.

[0049] Specifically, in this embodiment of the application, the InfoNCE loss function is used, and the formula is as follows: ,in The inner product of the mapped features measures semantic similarity. For temperature parameters, control the smoothness of the distribution (smaller). (Making the model more sensitive to hard-to-bear samples) Set the batch size to 256.

[0050] InfoNCE loss only focuses on the relative relationship between sample pairs and cannot guarantee the consistency of two modalities at the overall distribution level. Therefore, maximum mean difference (MMD) is introduced to explicitly constrain feature distribution alignment.

[0051] MMD is a nonparametric distribution distance metric based on kernel methods. Its core idea is that if two distributions are identical, then their mean representations in any kernel space should also be identical. Given two distributions P and Q, MMD is defined as: ,in This is the kernel mapping function that maps data to the Reproducing Kernel Hilbert Space (RKHS), where P and Q are the distributions of image features and text features, respectively. Since the expectation cannot be directly calculated, an empirical distribution is used for estimation, and the formula is: ,in For RBF kernel mapping, To share dimensional space, For kernel bandwidth parameters: The RBF kernel is a universal kernel that can theoretically approximate any continuous function in RKHS, thus enabling it to capture complex distribution differences. Furthermore, the decay characteristics of the RBF kernel make it sensitive to local structural changes in the feature space, allowing it to detect fine-grained distribution inconsistencies.

[0052] It should be noted that MMD can measure the distance between two distributions in the kernel space, thus ensuring that image features and text features follow the same distribution in a statistical sense. That is, the manifold distribution of the image and the Gaussian distribution of the text are mapped to a unified distribution form. In practice, after adding this loss, the FP during retrieval is significantly reduced.

[0053] In practical implementation, MMD can be expanded into the sum of three terms: ,in It's a kernel function; for the RBF kernel: .

[0054] In this expanded form, the first term measures the similarity structure within image features, the second term measures the similarity structure within text features, and the third term measures the cross-similarity between image and text features; when the distributions of the two modalities are completely identical, The greater the distribution difference, the greater the loss value.

[0055] Both InfoNCE and distribution alignment impose constraints in the target space, but excessive compression during the mapping process can lead to the loss of semantic information from the original features. This means all features are mapped to similar regions, causing model collapse. To address this, we use C-loss to ensure that the mapped features retain their original semantics: L ,in for divergence, and These are the probability distributions of the original features and the mapped features, respectively. By calculating the k-nearest neighbor distributions in the original feature space and the mapped feature space, the KL divergence of the two distributions is measured to ensure that the local neighborhood structure remains consistent before and after mapping. That is, if two images are semantically similar in the BLIP-2 space, they should also be similar in the mapped space.

[0056] Therefore, the loss for multi-objective joint optimization is: L .

[0057] S105. Based on the aligned image alignment features and text alignment features, generate a query vector to perform a query in the index database based on the query vector.

[0058] Specifically, for all multimodal documents in the document library to be retrieved, the document parsing and element separation steps, the heterogeneous feature extraction step, and the shared semantic space mapping step are executed in sequence to generate the corresponding image alignment feature or text alignment feature for each document element and build an index library.

[0059] During the real-time query phase, user queries are received. If the query is an image, its original image features are extracted using a visual encoder and input into a pre-trained image mapping network to obtain a query vector. If the query is text, its original text features are extracted using a text encoder and input into a pre-trained text mapping network to obtain a query vector. In the shared semantic space, the similarity between the query vector and all image alignment features and text alignment features in the index is calculated. The search results are then sorted according to the similarity, and the top-ranked document elements are returned as the final search results.

[0060] In this embodiment, local textual context and global domain knowledge are also introduced to guide VLM in generating higher-quality descriptions that better meet retrieval requirements. The formula is: ,in It is a local context window with k text blocks before and after it, extracting local context related to the image from the document; This provides global knowledge and includes a set of domain terms. Together with the business rule base R, contextual and domain knowledge are encoded into structured prompt words, and the optimal description is selected through multi-candidate generation and automatic scoring. These are the parameters for the pre-trained VLM.

[0061] The technical solution disclosed in this application significantly improves the semantic matching accuracy of image and text features by constructing a unified shared semantic space and using multi-objective joint optimization to drive feature alignment, ensuring that the original information of non-text elements is completely preserved in the feature mapping process, thereby achieving more accurate and relevant cross-modal retrieval results in intelligent retrieval applications.

[0062] Furthermore, the feature alignment mechanism reduces the reliance on large-scale pre-trained models and a large amount of labeled data, lowers the consumption of computing resources, and improves the practicality and scalability of the system. It can be flexibly applied to various business scenarios, such as technical document retrieval and multimodal question answering systems, providing reliable technical support for the efficient processing and intelligent application of heterogeneous data.

[0063] like Figure 2 As shown in the embodiments of this application, a multimodal feature alignment device for heterogeneous data is also proposed, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a multimodal feature alignment method for heterogeneous data as described in any of the above embodiments.

[0064] This application also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as: a multimodal feature alignment method for heterogeneous data as described in any of the above embodiments.

[0065] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0066] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0072] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0073] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0074] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0075] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.< / cell> < / row> ”、"

Claims

1. A method for aligning multimodal features of heterogeneous data, characterized in that, include: The system parses the input multimodal document to identify text and non-text elements. The non-text elements include natural image data and graphical data with logical structures; The text feature extraction engine extracts the original text features of the text elements, and based on the information carrying type of the non-text elements, the corresponding non-text feature extraction engine is called to extract the original image features of the natural image data and the original structured text features of the logically structured graphic data, respectively. A shared semantic space is constructed, and the original features of the image are mapped to the shared semantic space through an image mapping network, and the original features of the text and the original features of the structured text are mapped to the shared semantic space through a text mapping network. Based on the mapping features in the shared semantic space, multi-objective joint optimization is performed through a predefined multi-objective optimization function to drive feature alignment; Before performing multi-objective joint optimization to drive feature alignment based on the mapping features in the shared semantic space using a predefined multi-objective optimization function, the process further includes: Obtain the image mapping network and text mapping network in the shared semantic space; Calculate the similarity between image mapping features and text mapping features in the shared semantic space to obtain a similarity matrix, and construct a first loss function based on the similarity matrix; Calculate the first spatial distribution mean of the image mapping features and the second spatial distribution mean of the text mapping features, determine the distribution difference between the first spatial distribution mean and the second spatial distribution mean, and construct a second loss function based on the distribution difference; Calculate the first degree of synchronization between the image mapping features and the original image features, and the second degree of synchronization between the text mapping features and the original text features and the original structured text features. Based on the first degree of synchronization and the second degree of synchronization, construct a third loss function. The first loss function, the second loss function, and the third loss function are combined according to preset weights to construct a multi-objective optimization function; Based on the aligned image alignment features and text alignment features, a query vector is generated to perform a query in the index.

2. The method for aligning multimodal features of heterogeneous data according to claim 1, characterized in that, The process of mapping the original image features to the shared semantic space via an image mapping network, and mapping the original text features and the original structured text features to the shared semantic space via a text mapping network, specifically includes: Based on the dimension parameters of the shared semantic space, the weight parameters of the image mapping network and the text mapping network are initialized; The original features of the image are input into the image mapping network, and the image mapping network sequentially performs a first linear transformation, a nonlinear activation, and a second linear transformation on the original features of the image, mapping them to the shared semantic space. The original text features and the original structured text features are input into the text mapping network, and the text mapping network sequentially performs the first linear transformation on the original text features and the original structured text features. The nonlinear activation and the second linear transformation are mapped to the shared semantic space.

3. The method for aligning multimodal features of heterogeneous data according to claim 2, characterized in that, The step of sequentially performing a first linear transformation, a nonlinear activation, and a second linear transformation on the original features of the image through the image mapping network specifically includes: The original features of the image are input into the first linear transformation layer of the image mapping network for linear transformation and bias superposition. The original features of the linearly transformed image are input into the nonlinear activation layer of the image mapping network, and the original features of the linearly transformed image are subjected to nonlinear transformation through the activation function; The original image features after nonlinear transformation are input into the second linear transformation layer of the image mapping network for linear transformation and bias superposition, so as to project the original image features onto the shared semantic space.

4. The method for aligning multimodal features of heterogeneous data according to claim 1, characterized in that, The step of performing multi-objective joint optimization based on the mapping features in the shared semantic space to drive feature alignment through a predefined multi-objective optimization function specifically includes: Based on the mapping features in the shared semantic space, the first loss function, the second loss function, and the third loss function are solved respectively to obtain the feature distribution loss value corresponding to the first loss function, the information fidelity loss value corresponding to the second loss function, and the semantic similarity loss value corresponding to the third loss function. Based on the preset weights, the feature distribution loss value, the information fidelity loss value, and the semantic similarity loss value are weighted and summed to obtain the total loss value; Based on the total loss value, the parameter gradients of the image mapping network and the text mapping network are calculated using the backpropagation algorithm; Based on the parameter gradient, the parameters of the image mapping network and the text mapping network are updated to minimize the total loss value, thereby driving the image mapping features and text mapping features to align in the shared semantic space.

5. The method for aligning multimodal features of heterogeneous data according to claim 4, characterized in that, The parameters of the image mapping network and the text mapping network are updated according to the parameter gradient. Based on the parameter gradient and the preset learning rate, the update amount of each parameter in the image mapping network and the text mapping network is determined; Adjust the weights and bias parameters of the image mapping network and the text mapping network according to the update amount; The parameter update process is executed iteratively until the change in the total loss value is less than a preset threshold or the preset number of iterations is reached.

6. The method for aligning multimodal features of heterogeneous data according to claim 1, characterized in that, The process of parsing the input multimodal document and identifying text elements and non-text elements specifically includes: The multimodal document is subjected to semantic region detection using a pre-trained layout analysis model to obtain multiple candidate regions; Visual and texture features of each candidate region are extracted, and the candidate regions are classified by a classifier based on the visual and texture features to obtain text regions and non-text regions. The non-text area is analyzed to identify natural image data and graphical data with logical structures; the graphical data with logical structures includes at least flowcharts and data tables.

7. The method for aligning multimodal features of heterogeneous data according to claim 6, characterized in that, Based on the information carrying type of the non-text elements, the corresponding non-text feature extraction engine is invoked to extract the original image features of the natural image data and the original structured text features of the logically structured graphic data, specifically including: The visual encoder is invoked to extract features from the natural image data to obtain the original image features; The text content and logical relationships in the logically structured graphic data are identified by the structural parsing algorithm, a structured text description is generated, and a text encoder is called to extract features from the structured text description to obtain the original features of the structured text.

8. A multimodal feature alignment device for heterogeneous data, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a multimodal feature alignment method for heterogeneous data as described in any one of claims 1 to 7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to execute a multimodal feature alignment method for heterogeneous data as described in any one of claims 1 to 7.

Images