Multi-modal data retrieval method and device, equipment and medium

Abstract

The invention relates to the technical field of natural language processing, can be applied to business system platforms such as financial science and technology, medical health and the like, and discloses a multi-modal data retrieval method, device, equipment and medium, and the method comprises the following steps: carrying out multi-modal fusion on pre-acquired multi-modal data to obtain fused data; performing multi-scale feature extraction on the fused data to obtain fused data features; constructing a cross-modal incidence matrix according to the fused data features, and performing coarse-grained attention entropy evaluation on the cross-modal incidence matrix to obtain an evaluation result; performing fine-grained attention alignment on the fused data features according to an evaluation result to obtain an aligned feature vector; and performing weighted matching on the aligned feature vectors through preset user historical preference parameters to obtain a target modal retrieval result. Cross-modal association is comprehensively captured through coarse granularity and fine granularity, and the consistency of cross-modal semantics is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a multimodal data retrieval method, device, equipment and medium. Background Art

[0002] Multimodal retrieval is an information retrieval technology that integrates multiple data modalities (such as images, text, audio, video, etc.). Its core goal is to achieve mutual query and matching between different types of data through cross-modal semantic associations. This technology is gradually becoming a research and application hotspot in many fields; healthcare and financial technology are important fields closely related to people's lives and economic development. The application of multimodal retrieval technology has brought new development opportunities to these two fields. For example, in the healthcare field, by integrating data from multiple modalities, more subtle case characteristics can be revealed, thereby improving the accuracy and efficiency of disease diagnosis; in the field of financial technology, by retrieving and analyzing customers' multimodal data, financial institutions can deeply understand customer needs and preferences and provide customers with personalized financial products and services.

[0003] However, traditional methods usually use a single-scale attention mechanism for cross-modal alignment, but such methods can only capture cross-modal associations at a fixed granularity and cannot adapt to the multi-scale semantic characteristics of cross-modal data.

[0004] Therefore, in the face of the growing demand for multimodal retrieval, current multimodal retrieval methods urgently need to be improved to solve the problem that existing methods can only capture cross-modal associations from a fixed granularity and cannot adapt to the multi-scale semantic characteristics of cross-modal data. Summary of the Invention

[0005] The present invention provides a multimodal data retrieval method, apparatus, device and medium, which comprehensively capture cross-modal associations through coarse-grained and fine-grained methods to improve cross-modal semantic consistency.

[0006] In a first aspect, a multimodal data retrieval method is provided, comprising:

[0007] Performing multimodal fusion on pre-acquired multimodal data to obtain fused data;

[0008] Performing multi-scale feature extraction on the fused data to obtain fused data features;

[0009] Constructing a cross-modal association matrix according to the fused data features, and performing a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result;

[0010] Performing fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector;

[0011] The aligned feature vectors are weighted matched using preset user history preference parameters to obtain target modality retrieval results.

[0012] In a second aspect, a multimodal data retrieval device is provided, comprising:

[0013] A fusion module is used to perform multimodal fusion on the pre-acquired multimodal data to obtain fused data;

[0014] An extraction module, configured to perform multi-scale feature extraction on the fused data to obtain fused data features;

[0015] A construction module, configured to construct a cross-modal correlation matrix based on the fused data features;

[0016] An evaluation module, configured to perform a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result;

[0017] an alignment module, configured to perform fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector;

[0018] The matching module is used to perform weighted matching on the aligned feature vectors using preset user history preference parameters to obtain a target modality retrieval result.

[0019] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multimodal data retrieval method when executing the computer program.

[0020] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned multimodal data retrieval method are implemented.

[0021] In the solution implemented by the above-mentioned multimodal data retrieval method, device, computer equipment and storage medium, information of different modal data (such as text, image, audio, etc.) is integrated to make up for the limitations of single modal information, and multi-scale feature extraction is performed on the fused data to obtain features of different levels and granularity; a cross-modal association matrix is ​​constructed to clarify the association relationship between different modal features. For example, in image and text retrieval, the correspondence between text vocabulary and image areas is determined to provide a basis for subsequent cross-modal operations; key feature areas are screened based on entropy evaluation to highlight important information and filter noise and irrelevant features; at the same time, on a coarse-grained basis, more precise feature alignment is achieved so that key information of different modalities accurately corresponds in the semantic space; and user personalized preferences are incorporated, and feature vectors are weighted according to the user's past behavior habits to make the retrieval results more in line with user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a schematic diagram of an application environment of a multimodal data retrieval method in one embodiment of the present invention;

[0024] Figure 2 This is a flow chart of a multimodal data retrieval method according to an embodiment of the present invention;

[0025] Figure 3 is a structural diagram of a multimodal data retrieval device in one embodiment of the present invention;

[0026] Figure 4 is a structural diagram of a computer device according to an embodiment of the present invention;

[0027] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0029] The embodiment of the present invention provides a multimodal data retrieval method that can be applied in the following situations: Figure 1In an application environment, the client communicates with the server through a network. The server can perform multimodal fusion on the pre-acquired multimodal data to obtain fused data; perform multi-scale feature extraction on the fused data to obtain fused data features; construct a cross-modal association matrix based on the fused data features, and perform coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result; perform fine-grained attention alignment on the fused data features based on the evaluation result to obtain an aligned feature vector; perform weighted matching on the aligned feature vector by using preset user historical preference parameters to obtain a target modality retrieval result, and feed the target modality retrieval result back to the client. The present invention provides a multimodal data retrieval device, which captures cross-modal associations through coarse-grained and fine-grained comprehensive capture of cross-modal associations for the target modality retrieval result business, thereby improving cross-modal semantic consistency. Among them, the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 2 As shown, Figure 2 A flowchart of a multimodal data retrieval method provided by an embodiment of the present invention includes the following steps:

[0031] S1. Perform multimodal fusion on pre-acquired multimodal data to obtain fused data.

[0032] In an embodiment of the present invention, multimodal fusion refers to the process of organically integrating information from different data modalities, eliminating semantic differences and information barriers between modalities through algorithms or models, and generating unified representation data with multi-source characteristics.

[0033] Specifically, pre-collected multimodal data such as medical record text, medical images, patient voice (medical scenarios) or financial transaction text, stock market images, user voice interaction data (financial scenarios) are fused through feature alignment, semantic mapping and other technologies to form "fused data" that can comprehensively reflect multi-dimensional information.

[0034] In specific medical and health scenarios, by integrating electrocardiograms (waveform data), blood test reports (numerical data) and patient consultation voice (audio data), it is possible to identify hidden associations that are difficult to detect with a single modality (such as the linkage between abnormal heart rate and changes in voice intonation), thereby assisting in early disease warning.

[0035] In fintech scenarios, news text (such as corporate public opinion), stock market K-line charts (visual modality) and financial report numerical data are integrated to predict stock fluctuation trends. For example, by combining the sentiment of a company's negative news text with the technical indicators of the stock price chart, the risk of stock price decline can be identified in advance.

[0036] In the embodiment of the present invention, performing multimodal fusion on pre-acquired multimodal data to obtain fused data includes:

[0037] Divide the pre-acquired multimodal data into textual data and visual data;

[0038] Normalizing the visual data to obtain normalized visual data, and performing word segmentation on the text data to obtain text segmentation;

[0039] The normalized visual data and the text segmentation are fused through a preset fusion strategy to obtain fused data.

[0040] In an embodiment of the present invention, the normalization processing refers to the standardization of visual data to eliminate the influence of data scale and distribution differences, and the fusion refers to the information integration of the normalized visual data and the segmented text data through a specific algorithm to generate fused data with both modal features and realize cross-modal semantic association.

[0041] Specifically, the data modality is determined by the data format or label, and the recognized text data and visual data are stored in independent data structures (such as text corpus, image database) to facilitate subsequent sub-modal processing flow calls; pixel normalization and feature-level normalization are performed on the visual data. Taking images as an example for pixel-level normalization, the image pixel values ​​are linearly mapped from the original range (such as 0-255) to a unified interval (such as 0-1 or -1 to 1), and the numerical range is adjusted. If it needs to conform to the normal distribution, the mean and standard deviation of the pixel values ​​are calculated and standardized; taking the visual feature vector as an example for feature-level normalization, the feature vector extracted from the image (such as the feature matrix output by CNN) is L2 normalized to make the vector length 1, ensuring that the features of different images are comparable in the vector space.

[0042] In detail, the dictionary matching method and statistical learning method are used to perform word segmentation on the text data, and then the normalized visual data and text segmentation are fused early, late and cross-modal mapping respectively. The early fusion is to directly splice the visual feature vector (such as the 1024-dimensional feature extracted by CNN of the image) and the text word embedding vector (such as the word vector after word segmentation, with a dimension of 300) into a 1324-dimensional fusion feature vector, which is input into the subsequent model to assign weights to the visual features and text features: for example, set to 0.6 and 0.4 according to the task requirements, and generate a unified representation through fusion feature = ω1×visual feature + ω2×text feature.

[0043] Furthermore, when late fusion is required, independent models (such as visual classification model and text classification model) are first trained for visual data and text data respectively to obtain their respective prediction results. The sub-modal decisions are fused through a voting mechanism (such as majority voting) or weighted average (such as visual model weight 0.7, text model weight 0.3) to generate the final result; during cross-modal mapping fusion, the visual features and text word vectors are mapped to the same semantic space through a neural network (such as Transformer), and then the attention mechanism is associated. The specific operation is to calculate the attention weights of the visual area and the text words. For example, a strong association is established between the "heart" area in the image and the "heart disease" keyword in the text to enhance the semantic alignment of the fused data.

[0044] In the embodiment of the present invention, the data is divided into text data and visual data to clarify the modal boundaries of the data, laying the foundation for fine-grained sub-modal processing. The scale differences of the visual data are eliminated through normalization processing to ensure feature comparability. At the same time, the fusion operation realizes cross-modal information complementarity through feature splicing, semantic mapping, etc., providing more comprehensive input for subsequent multimodal analysis.

[0045] In the embodiments of the present invention, information aggregation is used to break through the limitations of a single modality, facilitate cross-modal information complementation, semantic association mining, and decision-making dimension expansion, thereby improving the efficiency and accuracy of problem solving in complex scenarios.

[0046] S2. Perform multi-scale feature extraction on the fused data to obtain fused data features.

[0047] In the embodiment of the present invention, the multi-scale feature extraction refers to the process of performing feature mining on data through different scales, such as windows of different sizes, network structures of different levels, and inputs of different resolutions.

[0048] Specifically, multi-scale feature extraction can be based on multi-scale extraction of deep learning, and the fused data features are extracted through different layers of convolutional neural networks (CNN): the shallow network captures small-scale features such as edges and textures, and the deep network captures large-scale features such as semantics and structures. Subsequently, a pyramid structure, such as spatial pyramid pooling or multi-branch networks, is used to process the input data at different resolutions, such as extracting features after scaling the image to different sizes.

[0049] In detail, visual input After processing by the four-level Swin Transformer block, it can be expressed as:

[0050] V c =Swin 1 / 16 (V),V f =Swin 1 / 4 (V)

[0051] in, It is a coarse-grained feature containing semantic embeddings of 16×16 pixel blocks; Preserve local detail features;

[0052] Furthermore, the text input T is encoded by Longformer and can be expressed as follows:

[0053] T c =AvgPool(Longformer(T)),T f =CLS(Longformer(T))

[0054] in, is the paragraph-level average pooling result, A fine-grained representation of the [CLS] tag.

[0055] In medical and health scenarios, medical image analysis can extract detailed features such as lesion edges and calcification points on a small scale, and analyze organ locations and the overall range of lesions on a large scale. Combining the two can reduce missed diagnoses, such as early-stage tiny tumors, and misdiagnoses, such as misjudging normal tissue as lesions.

[0056] In specific scenarios of financial technology, transaction data features are extracted, and the real-time features of single transactions, such as amount, time, IP address, etc., are analyzed on a small scale. Historical transaction patterns of users, such as monthly consumption habits and cross-account fund flows, are analyzed on a large scale to identify fraudulent transactions. For example, small-amount, high-frequency abnormal transfers may show regular patterns at a large scale.

[0057] In the embodiment of the present invention, the multi-scale feature extraction of the fused data to obtain fused data features includes:

[0058] Performing multi-level text embedding on the text data in the fused data to obtain text features;

[0059] Performing multi-resolution image feature extraction on the visual data in the fused data to obtain visual features;

[0060] The text features and the visual features are combined to generate fused data features.

[0061] In an embodiment of the present invention, the multi-level text embedding refers to the process of gradually mapping text data from the original character / word level to semantic spaces at different abstract levels. The multi-resolution image feature extraction refers to feature extraction of image data at different pixel resolutions, fusing details and global information at different scales to avoid information loss at a single resolution. The combined generation refers to fusing text features and visual features through a specific strategy to form a unified multimodal feature representation.

[0062] Specifically, multi-level text embedding is performed on the text data in the fusion data, which can be divided into low-level feature extraction, mid-level feature extraction, and high-level feature extraction. The low-level feature extraction can be achieved through character-level embedding and word-level embedding. Character-level embedding is to split the text into single characters, generate character vectors through the character embedding matrix, and capture low-level features such as spelling and affixes. Word-level embedding is to split the text into words through word segmentation tools, and use models such as Word2Vec and GloVe to generate word vectors to retain the semantic information of words; mid-level feature extraction is to capture the order relationship of words through recurrent neural networks or Transformer, generate phrase-level features, and use syntactic analysis to extract sentence structure features; high-level feature extraction is to use a pre-trained language model to encode the entire sentence or paragraph, and generate context-sensitive semantic vectors based on the context.

[0063] Furthermore, multi-resolution image feature extraction refers to scaling the original image to different sizes (such as original image, 1 / 2 size, 1 / 4 size) to form an image pyramid, which can be defined as image resolution adjustment, followed by feature extraction at different resolutions. High-resolution feature extraction uses shallow CNN to extract edges, textures and other details from the original image, and low-resolution feature extraction uses deep CNN to extract semantic-level features from downsampled images, such as the overall morphology of the lungs in medical images and the long-term trend morphology of financial candlestick charts.

[0064] Furthermore, the combined generation of fused data features requires feature alignment and dimensionality unification first, that is, dimension mapping of text features and visual features (such as through a fully connected layer) so that they are in the same dimensional space. The combination strategies can be early fusion, late fusion, and attention fusion. Early fusion is to directly splice text and visual features in the feature extraction stage, such as splicing the text embedding vector with the image feature map and inputting it into the subsequent network. Late fusion is to process text and visual features independently, and then fuse them through weighted summation, attention mechanism, etc. Attention fusion is through cross-modal attention mechanism (such as text features guiding the extraction of key areas of visual features). For example, in a medical scenario, the text mentions "left upper lobe nodule", and the attention mechanism will enhance the visual feature weight of the corresponding area; the feature expression ability is enhanced through nonlinear transformation or regularization, and finally a fused feature vector containing text semantics and visual information is generated.

[0065] In the embodiment of the present invention, multi-level text embedding achieves a deep understanding of text semantics through hierarchical extraction of characters, words, and semantics; multi-resolution image features capture details and global information at different scales; and combined generation uses feature alignment and fusion strategies to complement multimodal information.

[0066] In the embodiment of the present invention, the adaptability and robustness of the model to complex scenarios are enhanced through the complementarity of multi-dimensional features.

[0067] S3. Construct a cross-modal association matrix based on the fused data features, and perform a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result.

[0068] In an embodiment of the present invention, the construction refers to establishing a correlation quantization matrix for the feature vectors of different modalities in the fusion data features, and the coarse-grained attention entropy value evaluation refers to evaluating the uncertainty or information concentration of the attention distribution in the cross-modal correlation matrix from a macro (coarse-grained) level, which is achieved through entropy value calculation.

[0069] Specifically, through similarity calculation (such as cosine similarity, dot product) or attention mechanism, the degree of correlation between each text feature and the visual feature is calculated to form a two-dimensional matrix. The correlation matrix is ​​aggregated by rows or columns to form the modal-level attention distribution. The entropy of the attention distribution is calculated using the information entropy formula (such as Shannon entropy). The higher the entropy value, the more dispersed the attention, and the lower the entropy value, the more concentrated the information interaction.

[0070] In specific medical and health scenarios, the correspondence between text descriptions and image lesions is located to assist doctors in verifying diagnostic consistency. For example, if the text mentions "liver tumor", the correlation between this word and the liver image area in the matrix should be high.

[0071] In financial scenarios, by combining financial report text (financial descriptions, risk warnings) with chart data (stock price trends, trading volume), we can build the correlation between financial report keywords and chart feature points, discover the potential correlation between text semantics and market fluctuations, and assist in the construction of quantitative investment strategies.

[0072] In an embodiment of the present invention, constructing a cross-modal association matrix based on the fused data features includes:

[0073] Projecting the text features and visual features in the fused data features into the same low-dimensional space to obtain a text embedding vector and a visual embedding vector;

[0074] Calculating the correlation between the text embedding vector and the visual embedding vector;

[0075] The calculated correlation results are converted into a matrix to obtain a cross-modal correlation matrix.

[0076] In an embodiment of the present invention, the projection into the same low-dimensional space refers to mapping high-dimensional feature vectors of different modalities into a low-dimensional space of the same dimension to eliminate the dimensional difference between the modalities. The calculation refers to quantifying the similarity or correlation between the text embedding vector and the visual embedding vector in the low-dimensional space. The transformation refers to organizing the correlation results of all text embedding vectors and visual embedding vectors into a two-dimensional matrix.

[0077] Specifically, the text features and visual features are reduced in dimensionality through matrix multiplication to ensure that the vectors of different modalities are "semantically comparable" in the low-dimensional space after projection. For example, the "heart" in the text and the features of the heart CT image are close in space after projection; then the cosine similarity of the text vector and the visual vector is calculated to characterize the correlation strength between the semantic and visual features.

[0078] Furthermore, assume that the text embedding vector set is T = {t1, t2, ... t m}, the visual embedding vector set is V = {v1,v2,...v n}, where m is the number of text features, n is the number of text features; the matrix size is m×n, element (i, j) = sim(t i ,v j ), which is the correlation between the i-th text vector and the j-th visual vector.

[0079] In the embodiment of the present invention, the coarse-grained attention entropy evaluation of the cross-modal association matrix to obtain the evaluation result includes:

[0080] Performing normalized attention distribution processing on the row elements in the cross-modal association matrix using a Softmax function to obtain an attention probability distribution matrix;

[0081] Calculating a local entropy value of the attention probability distribution matrix, and performing statistical aggregation on the attention probability distribution matrix according to the local entropy value to obtain an aggregation result;

[0082] An entropy evaluation is performed on the cross-modal association matrix according to the aggregation result to obtain an evaluation result.

[0083] In an embodiment of the present invention, the normalized attention distribution processing refers to normalizing the row elements of the cross-modal association matrix through the Softmax function, so that each row element is converted into a probability distribution, which is used to represent the attention weight between different modal features. The calculation of the local entropy value refers to measuring the uncertainty or degree of dispersion of the attention distribution. The higher the entropy value, the more dispersed the attention, and vice versa. The statistical aggregation refers to aggregating the local entropy values ​​of each row into an overall indicator through statistical methods. The entropy value evaluation refers to the process of quantitatively judging the feature correlation and attention quality of the cross-modal association matrix based on the aggregated entropy value indicator.

[0084] Specifically, assuming that the cross-modal correlation matrix is ​​M, with a size of m×n, for the i-th row element in the matrix [M i1 ,M i2 ,...,M in ] Apply the Softmax function, which is expressed as follows:

[0085]

[0086] Finally, the result is the attention probability distribution matrix A.

[0087] Furthermore, for each row of the attention probability distribution matrix A, [A i1 ,A i2 ,...,A in ], calculate the information entropy, the specific formula is as follows:

[0088]

[0089] Among them, A ij represents the probability distribution value of the i-th row and j-th column, and H(i) represents the entropy value of the i-th row of the attention probability distribution matrix A.

[0090] Furthermore, assuming that m local entropy values ​​[H(1),H(2),....,H(m)] are obtained, weighted aggregation is used to assign weights ω according to feature importance. i , the specific formula is as follows:

[0091]

[0092] Among them, Agg represents the aggregation result.

[0093] Furthermore, an evaluation threshold or standard is set, and the aggregation result is mapped into an evaluation conclusion. If Agg is high, it means that the cross-modal feature correlation is scattered, and there may be noise or semantic mismatch; if Agg is low, it means that the attention is concentrated and the cross-modal feature correlation is strong.

[0094] In the embodiments of the present invention, cross-modal feature association, attention weight normalization, entropy quantification dispersion, and aggregation evaluation correlation are used to assist accurate diagnosis in medical care and support intelligent decision-making in finance. Entropy evaluation is used to filter noise, strengthen effective associations, and improve the utilization efficiency of cross-modal data.

[0095] In an embodiment of the present invention, by constructing a cross-modal association matrix and entropy value evaluation, multimodal data is upgraded from "feature fusion" to "interactive quantification", providing a complete link from data integration to decision support for the medical and financial fields.

[0096] S4. Perform fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector.

[0097] In an embodiment of the present invention, the fine-grained attention alignment refers to a more refined adjustment of the attention weights of cross-modal features based on the entropy evaluation results, so as to achieve precise semantic and structural matching between text and visual features, and finally generate a fused alignment feature vector.

[0098] Specifically, based on the entropy evaluation results (such as low entropy areas indicate strong associations, and high entropy areas indicate weak associations or noise), the attention weights of the cross-modal association matrix are dynamically adjusted, the weight coefficients of the association weights corresponding to low entropy values ​​are increased, and the weights corresponding to high entropy values ​​are suppressed; the corrected weights are used to perform a weighted summation of text features and visual features, and the final output vector integrates precise cross-modal alignment information.

[0099] In the embodiment of the present invention, performing fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector includes:

[0100] Screening the cross-modal association matrix for key feature regions according to the evaluation result and the fused data features to obtain screened feature regions;

[0101] Performing pixel-word matching on the screened feature area to obtain a pixel-word association matrix;

[0102] The pixel-word association matrix is ​​subjected to local consistency constraints according to a preset spatial continuity loss function to obtain an aligned feature vector.

[0103] In an embodiment of the present invention, the key feature area screening refers to identifying and extracting the most discriminative local area for the task from the cross-modal association matrix based on the entropy evaluation results and the fusion data features. The pixel-word matching refers to semantically associating the pixel-level features of the visual data with the words after text segmentation, constructing a pixel-word correspondence matrix, and realizing fine-grained alignment of the visual space and the text semantics. The local consistency constraint refers to constraining the pixel-word association matrix based on a preset spatial continuity loss function to ensure that the word-word matching results of adjacent pixels are semantically consistent.

[0104] Specifically, the entropy evaluation result (such as the entropy value of the attention probability distribution matrix) is superimposed with the importance score of the fusion data feature (such as the modulus of the feature vector) to generate a comprehensive weight matrix W, set the threshold τ, and retain W ij The region with τ > τ is used to form candidate key regions, and morphological processing (such as dilation and erosion) is performed on the candidate regions to ensure that the key regions form connected blocks in the matrix and avoid interference from scattered feature points.

[0105] Furthermore, the text features and visual features in the filtered feature area are subjected to feature dimensionality reduction and projection, and then the cosine similarity is calculated for each pixel and word to generate a pixel-word association matrix. Then, based on the preset spatial continuity loss function, the pixel-word association matrix is ​​constrained to ensure that the word-word matching results of adjacent pixels are semantically consistent, avoiding local contradictions in cross-modal mapping (such as adjacent pixels being matched to words with opposite semantics).

[0106] Specifically, the commonly used spatial continuity loss function is as follows:

[0107] L smooth =∑ (p,p′)∈neighbor (M p,w -M p′,w ) 2

[0108] Where (p, p′) represents adjacent pixels, p represents pixel, and w represents word unit. The matching probability difference between adjacent pixels for the same word unit is constrained to be minimized.

[0109] Furthermore, the loss function is minimized by gradient descent and other methods to optimize the pixel-to-word association matrix M′. Finally, the alignment feature vector is generated by weighted summation. The formula is as follows:

[0110] Aligned feature vector = ∑ p,w M′ p,w ·(V p +T w )

[0111] Among them, V p Represents the feature vector of each pixel, Tw For example, the word vector is generated by embedding the word unit after word segmentation.

[0112] In an embodiment of the present invention, a closed loop of screening, matching, and constraints is used to upgrade cross-modal data from feature fusion to semantic alignment. Through spatial continuity and semantic consistency constraints, the local ambiguity problem in cross-modal mapping is resolved, further improving the domain adaptability of the model.

[0113] In an embodiment of the present invention, fine-grained attention alignment is a feature optimization mechanism based on evaluation feedback, which guides the local weight adjustment of cross-modal features through the entropy evaluation results, thereby achieving an upgrade from global association to local precise mapping.

[0114] S5. Perform weighted matching on the aligned feature vectors using preset user history preference parameters to obtain a target modality retrieval result.

[0115] In an embodiment of the present invention, the weighted matching refers to assigning differentiated weights to features of different dimensions in the aligned feature vector according to preset user historical preference parameters, and achieving retrieval and matching of target modalities (such as text and images) through weighted calculation.

[0116] Specifically, through reinforcement learning or incremental learning, the weights are dynamically adjusted according to user interaction behaviors (such as clicks, favorites, and retrieval result feedback), and each dimension of the aligned feature vector is corresponded to the specific dimension of user preference. For each candidate sample in the target modality database, its weighted matching score with the aligned feature vector is calculated, and the retrieval results are obtained by arranging them in descending order of the scores.

[0117] In an embodiment of the present invention, performing weighted matching on the aligned feature vectors using preset user history preference parameters to obtain a target modality retrieval result includes:

[0118] Parsing the preset user history preference parameters to obtain parsed preference parameters;

[0119] Performing weight matrix matching conversion on the parsing preference parameters according to the aligned feature vector to obtain a user preference weight matrix;

[0120] Performing dimension-by-dimension weighted calculation on the user preference weight matrix and the alignment feature vector to obtain a weighted value;

[0121] The target modality retrieval results are filtered according to the weighted value.

[0122] In an embodiment of the present invention, the parsing refers to parsing the implicit preference information in the user's historical interaction data (such as click records, favorite tags) into computable numerical parameters, the weight matrix matching conversion refers to mapping the parsed one-dimensional preference parameters into a two-dimensional weight matrix that matches the dimension of the alignment feature vector, so as to achieve alignment of user preferences with cross-modal features, the dimension-by-dimension weighted calculation refers to multiplying the weight matrix and the alignment feature vector element by element, strengthening the feature dimension of the user preference and suppressing irrelevant dimensions, and the screening refers to sorting and screening the candidate modal data according to the weighted value, and outputting the target result that meets the user preference.

[0123] Specifically, the preset user historical preference parameters are cleaned and features are extracted, and then the parameters are quantified and standardized to convert qualitative preferences into numerical weights. For example, the doctor's attention to the pathological type is mapped to a weight value in the interval [0,1]. For example, the frequency of "pulmonary nodules" in user historical queries is 80%, and "mediastinal widening" is 20%, which is parsed into a weight vector [0.8,0.2].

[0124] Specifically, assuming that the dimension of the aligned feature vector is n and the dimension of the user preference parameter is m, the preference parameter is expanded into a weight vector of the same dimension as the feature vector through the mapping function f:m→n, generating a weight matrix W∈R^(m×n), which represents the influence weight of the i-th preference parameter on the j-th feature dimension; calculating the weighted value vector v=W×feat, where feat is the aligned feature vector; calculating the similarity between the weighted feature vector and each sample in the candidate modality library, setting a similarity threshold, retaining samples with a similarity greater than the similarity threshold, and arranging them in descending order by score, and finally obtaining the target modality retrieval result.

[0125] In the embodiment of the present invention, by converting user historical preferences into feature weights, a bridge is built between universal cross-modal features and personalized needs, and the output is more in line with the user's actual needs.

[0126] It can be seen that in the above scheme, for the target modality retrieval result business, multimodal fusion is performed on the pre-acquired multimodal data to obtain fused data; multi-scale feature extraction is performed on the fused data to obtain fused data features; a cross-modal association matrix is ​​constructed based on the fused data features, and a coarse-grained attention entropy evaluation is performed on the cross-modal association matrix to obtain an evaluation result; fine-grained attention alignment is performed on the fused data features based on the evaluation result to obtain an aligned feature vector; weighted matching is performed on the aligned feature vector using preset user historical preference parameters to obtain the target modality retrieval result, and cross-modal associations are captured through coarse-grained and fine-grained comprehensive methods to improve cross-modal semantic consistency.

[0127] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0128] In one embodiment, a multimodal data retrieval device is provided, and the multimodal data retrieval device corresponds one-to-one to a multimodal data retrieval method in the above embodiment. Figure 3 As shown, the multimodal data retrieval device includes a fusion module 101, an extraction module 102, a construction module 103, an evaluation module 104, an alignment module 105, and a matching module 106. The functional modules are described in detail as follows:

[0129] The fusion module 101 is used to perform multimodal fusion on the pre-acquired multimodal data to obtain fused data;

[0130] An extraction module 102 is configured to perform multi-scale feature extraction on the fused data to obtain fused data features;

[0131] A construction module 103 is used to construct a cross-modal correlation matrix based on the fused data features;

[0132] An evaluation module 104 is configured to perform a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result;

[0133] An alignment module 105 is configured to perform fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector;

[0134] The matching module 106 is configured to perform weighted matching on the aligned feature vectors using preset user history preference parameters to obtain a target modality retrieval result.

[0135] In one embodiment, the fusion module 101, when performing multimodal fusion on pre-acquired multimodal data to obtain fused data, is configured to:

[0136] Divide the pre-acquired multimodal data into textual data and visual data;

[0137] Normalizing the visual data to obtain normalized visual data, and performing word segmentation on the text data to obtain text segmentation;

[0138] The normalized visual data and the text segmentation are fused through a preset fusion strategy to obtain fused data.

[0139] In one embodiment, when the extraction module 102 performs multi-scale feature extraction on the fused data to obtain fused data features, it is configured to:

[0140] Performing multi-level text embedding on the text data in the fused data to obtain text features;

[0141] Performing multi-resolution image feature extraction on the visual data in the fused data to obtain visual features;

[0142] The text features and the visual features are combined to generate fused data features.

[0143] In one embodiment, when constructing the cross-modal association matrix based on the fused data features, the construction module 103 is configured to:

[0144] Projecting the text features and visual features in the fused data features into the same low-dimensional space to obtain a text embedding vector and a visual embedding vector;

[0145] Calculating the correlation between the text embedding vector and the visual embedding vector;

[0146] The calculated correlation results are converted into a matrix to obtain a cross-modal correlation matrix.

[0147] In one embodiment, when the evaluation module 104 performs a coarse-grained attention entropy evaluation on the cross-modal association matrix and obtains an evaluation result, it is configured to:

[0148] Performing normalized attention distribution processing on the row elements in the cross-modal association matrix using a Softmax function to obtain an attention probability distribution matrix;

[0149] Calculating a local entropy value of the attention probability distribution matrix, and performing statistical aggregation on the attention probability distribution matrix according to the local entropy value to obtain an aggregation result;

[0150] An entropy evaluation is performed on the cross-modal association matrix according to the aggregation result to obtain an evaluation result.

[0151] In one embodiment, when the alignment module 105 performs fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector, it is configured to:

[0152] Screening the cross-modal association matrix for key feature regions according to the evaluation result and the fused data features to obtain screened feature regions;

[0153] Performing pixel-word matching on the screened feature area to obtain a pixel-word association matrix;

[0154] The pixel-word association matrix is ​​subjected to local consistency constraints according to a preset spatial continuity loss function to obtain an aligned feature vector.

[0155] In one embodiment, when performing weighted matching on the aligned feature vectors using preset user history preference parameters to obtain a target modality retrieval result, the matching module 106 is configured to:

[0156] Parsing the preset user history preference parameters to obtain parsed preference parameters;

[0157] Performing weight matrix matching conversion on the parsing preference parameters according to the aligned feature vector to obtain a user preference weight matrix;

[0158] Performing dimension-by-dimension weighted calculation on the user preference weight matrix and the alignment feature vector to obtain a weighted value;

[0159] The target modality retrieval results are filtered according to the weighted value.

[0160] The present invention provides a multimodal data retrieval device. For a target modality retrieval result service, the device performs multimodal fusion on pre-acquired multimodal data to obtain fused data; performs multi-scale feature extraction on the fused data to obtain fused data features; constructs a cross-modal association matrix based on the fused data features, and performs coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result; performs fine-grained attention alignment on the fused data features based on the evaluation result to obtain an aligned feature vector; and performs weighted matching on the aligned feature vector using preset user historical preference parameters to obtain a target modality retrieval result. The device captures cross-modal associations through coarse-grained and fine-grained comprehensive capture, thereby improving cross-modal semantic consistency.

[0161] For the specific definition of a multimodal data retrieval device, please refer to the definition of a multimodal data retrieval method above, and will not be repeated here. The various modules in the above-mentioned multimodal data retrieval device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a multimodal data retrieval method.

[0163] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a multimodal data retrieval method.

[0164] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0165] Performing multimodal fusion on pre-acquired multimodal data to obtain fused data;

[0166] Performing multi-scale feature extraction on the fused data to obtain fused data features;

[0167] Constructing a cross-modal association matrix according to the fused data features, and performing a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result;

[0168] Performing fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector;

[0169] The aligned feature vectors are weighted matched using preset user history preference parameters to obtain target modality retrieval results.

[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0171] Performing multimodal fusion on pre-acquired multimodal data to obtain fused data;

[0172] Performing multi-scale feature extraction on the fused data to obtain fused data features;

[0173] Constructing a cross-modal association matrix according to the fused data features, and performing a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result;

[0174] Performing fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector;

[0175] The aligned feature vectors are weighted matched using preset user history preference parameters to obtain target modality retrieval results.

[0176] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0177] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0178] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0179] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. If software tools or components other than those of the company appear in the application embodiments, they are merely used for illustration and do not represent actual use. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A multimodal data retrieval method, characterized in that: include: Performing multimodal fusion on pre-acquired multimodal data to obtain fused data; Performing multi-scale feature extraction on the fused data to obtain fused data features; Constructing a cross-modal association matrix according to the fused data features, and performing a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result; Performing fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector; The aligned feature vectors are weighted matched using preset user history preference parameters to obtain target modality retrieval results.

2. The multimodal data retrieval method according to claim 1, wherein: The performing multimodal fusion on the pre-acquired multimodal data to obtain fused data includes: Divide the pre-acquired multimodal data into textual data and visual data; Normalizing the visual data to obtain normalized visual data, and performing word segmentation on the text data to obtain text segmentation; The normalized visual data and the text segmentation are fused through a preset fusion strategy to obtain fused data.

3. The multimodal data retrieval method according to claim 1, wherein: The performing multi-scale feature extraction on the fused data to obtain fused data features includes: Performing multi-level text embedding on the text data in the fused data to obtain text features; Performing multi-resolution image feature extraction on the visual data in the fused data to obtain visual features; The text features and the visual features are combined to generate fused data features.

4. The multimodal data retrieval method according to claim 1, wherein: The constructing a cross-modal association matrix according to the fused data features includes: Projecting the text features and visual features in the fused data features into the same low-dimensional space to obtain a text embedding vector and a visual embedding vector; Calculating the correlation between the text embedding vector and the visual embedding vector; The calculated correlation results are converted into a matrix to obtain a cross-modal correlation matrix.

5. The multimodal data retrieval method according to claim 1, wherein: The coarse-grained attention entropy evaluation of the cross-modal association matrix is ​​performed to obtain an evaluation result, including: Performing normalized attention distribution processing on the row elements in the cross-modal association matrix using a Softmax function to obtain an attention probability distribution matrix; Calculating a local entropy value of the attention probability distribution matrix, and performing statistical aggregation on the attention probability distribution matrix according to the local entropy value to obtain an aggregation result; An entropy evaluation is performed on the cross-modal association matrix according to the aggregation result to obtain an evaluation result.

6. The multimodal data retrieval method according to claim 1, wherein: The performing fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector includes: Screening the cross-modal association matrix for key feature regions according to the evaluation result and the fused data features to obtain screened feature regions; Performing pixel-word matching on the screened feature area to obtain a pixel-word association matrix; The pixel-word association matrix is ​​subjected to local consistency constraints according to a preset spatial continuity loss function to obtain an aligned feature vector.

7. The multimodal data retrieval method according to claim 1, wherein: The weighted matching of the aligned feature vectors by using preset user history preference parameters to obtain target modality retrieval results includes: Parsing the preset user history preference parameters to obtain parsed preference parameters; Performing weight matrix matching conversion on the parsing preference parameters according to the aligned feature vector to obtain a user preference weight matrix; Performing dimension-by-dimension weighted calculation on the user preference weight matrix and the alignment feature vector to obtain a weighted value; The target modality retrieval results are filtered according to the weighted value.

8. A multimodal data retrieval device, characterized in that: include: A fusion module is used to perform multimodal fusion on the pre-acquired multimodal data to obtain fused data; An extraction module, configured to perform multi-scale feature extraction on the fused data to obtain fused data features; A construction module, configured to construct a cross-modal correlation matrix based on the fused data features; An evaluation module, configured to perform a coarse-grained attention entropy evaluation on the cross-modal association matrix to obtain an evaluation result; an alignment module, configured to perform fine-grained attention alignment on the fused data features according to the evaluation results to obtain an aligned feature vector; The matching module is used to perform weighted matching on the aligned feature vectors using preset user history preference parameters to obtain a target modality retrieval result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal data retrieval method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal data retrieval method according to any one of claims 1 to 7 is implemented.

Images