A cross-modal association recognition system for heterogeneous data understanding

Abstract

The application discloses a cross-modal correlation identification system for heterogeneous data understanding and belongs to the field of data processing.The cross-modal correlation identification system for heterogeneous data understanding comprises a first data acquisition module which is used for receiving an input specific application scene of cross-modal correlation identification and involved heterogeneous modalities, and collecting data of different modalities based on the specific application scene.Compared with the prior art, the cross-modal correlation identification system has the beneficial effects that: the cross-modal correlation identification system collects data and processes the data by clearly defining the specific application scene of cross-modal correlation identification and the involved heterogeneous modalities, realizes cross-modal correlation by sharing the distance measure in the semantic space, has high information retrieval efficiency, and the designed modal conflict detection and analysis module can effectively identify conflict scenes, avoids the situation that the standard cross-modal correlation process is invalid due to the default semantic consistency between modalities in the multi-modal information conflict scene, and the designed semantic calibration module further analyzes the conflict scene to determine whether it is reasonable.

Description

Technical Field

[0001] This invention belongs to the field of data processing, and in particular relates to a cross-modal association recognition system for understanding heterogeneous data. Background Technology

[0002] In the current technological environment, the integration of heterogeneous data (such as text, images, and videos) still heavily relies on manual operation, resulting in low efficiency and high costs. Without effective cross-modal association technologies, different modalities will form information silos due to semantic barriers. A typical example is the inability to automatically associate video content with accompanying text descriptions, leading to low information retrieval efficiency. More seriously, single-modal analysis can lose cross-modal collaborative information. For instance, when the emotional tone of voice in a video is inconsistent with the semantic meaning of actions on screen, manual review has a high error rate, directly impacting the decision-making accuracy of intelligent systems. These pain points urgently need to be addressed through automated cross-modal association technologies. Summary of the Invention

[0003] Therefore, it is necessary to provide a cross-modal association recognition system for understanding heterogeneous data in order to address the above problems.

[0004] The present invention is implemented as follows: a cross-modal association recognition system for heterogeneous data understanding includes:

[0005] The first data acquisition module is used to receive the specific application scenario of cross-modal association recognition (such as image-text matching, video question answering, or multimodal sentiment analysis) and the heterogeneous modalities involved (such as text, image, and audio) based on the specific application scenario, collect data from different modalities, and ensure that the collected data from different modalities are aligned semantically and spatiotemporally (i.e., they describe or perceive the same entity or event), thereby obtaining the first data of different modalities.

[0006] The second data acquisition module is used to independently perform unique cleaning, standardization, and transformation operations (such as text segmentation to remove stop words, image size normalization, and audio resampling and frame division) on the first data of each modality to obtain the second data of each modality. Features are extracted from the second data of each modality (such as text processing by BERT, image processing by ResNet, and audio processing by MFCCs; BERT is a speech model, ResNet is a convolutional neural network architecture, and MFCCs is an audio feature extraction method). The extracted features can capture the core semantic information of the modality.

[0007] The model design module is used to design the multimodal model architecture (including each modal sub-network + shared mapping network), inputting the extracted heterogeneous features into the model; by optimizing specific loss functions (such as contrastive loss and triplet loss), it drives the representation distance of semantically related cross-modal sample pairs (i.e., aligned samples) in the shared semantic space to converge (measured by cosine similarity / Euclidean distance), while pushing away irrelevant sample pairs, thereby learning cross-modal association representations;

[0008] The cross-modal association module is used to map the data to be identified to a shared semantic space using the trained model. It directly implements cross-modal association tasks based on distance metrics (cosine similarity / Euclidean distance) in the space. The cross-modal association tasks include cross-modal retrieval (single-modal query → cross-modal result return), matching (cross-modal sample pair relevance determination), and generation (cross-modal conditional content generation).

[0009] In one embodiment, the present invention provides a cross-modal association recognition system for understanding heterogeneous data, wherein the first data acquisition module includes:

[0010] The requirement parsing unit is used to receive specific application scenarios for cross-modal association recognition (such as image-text matching, video question answering, or multimodal sentiment analysis) and the heterogeneous modalities involved (such as text, image, and audio), and parse the scenario requirements and modal interaction logic.

[0011] The data acquisition and processing unit is used to collect raw data based on scenario requirements. It achieves cross-modal semantic / spatiotemporal alignment through timestamp synchronization (aligning video frames and audio segments) or metadata binding (such as associating image IDs with descriptive text), ensuring that different modal data describe the same entity or event, and obtaining the first data with structured alignment.

[0012] In one embodiment, the present invention provides a cross-modal association recognition system for heterogeneous data understanding, wherein the second data acquisition module includes:

[0013] The cleaning and standardization unit is used to independently perform unique cleaning and standardization on the first data of each modality. It performs word segmentation, removal of stop words and punctuation marks on text data; it uniformly scales image data to 224×224 resolution and normalizes pixel values ​​to [0,1]; and it resamples audio data to 16kHz and then performs frame processing (frame length 25ms, frame shift 10ms).

[0014] The format normalization unit is used to convert the cleaned and normalized first data into a normalized format that the model can process, i.e., the second data. Text data is converted into a word index sequence (mapped to an integer sequence through a vocabulary), image data is converted into a three-channel RGB tensor (3×224×224 dimension), and audio data is converted into a Mel spectrogram (generated as a time-frequency matrix through short-time Fourier transform).

[0015] The feature extraction unit is used to extract features using a pre-trained model. Text data is input into the index sequence to the BERT model (a pre-trained language model) to obtain a 768-dimensional semantic vector of the [CLS] flag; image data is input into the RGB tensor to the ResNet50 (a deep convolutional neural network model) to extract 2048-dimensional features of the final pooling layer; audio data is processed by the MFCCs algorithm (an audio feature extraction method) to calculate a 40-dimensional time-frequency feature matrix for each frame.

[0016] In one embodiment, the present invention provides a cross-modal association recognition system for heterogeneous data understanding, wherein the model design module includes:

[0017] The model design unit is used to design multi-branch network architectures. The text branch consists of a BERT layer with frozen parameters and a trainable BiLSTM layer (bidirectional long short-term memory network). The image branch consists of a ResNet50 convolutional backbone network and an adaptive pooling layer. The shared mapping layer is a two-layer fully connected network (ReLU activation function, outputting a 128-dimensional vector).

[0018] The model training unit is used to optimize the triplet loss using an online hard sample mining strategy: each training session randomly selects an anchor sample (such as an image), matches positive samples with the same ID (their corresponding text descriptions) and negative samples with different IDs (other texts), and forces the Euclidean distance between the anchor point and the positive sample to be at least 0.3 smaller than the distance between the anchor point and the negative sample through backpropagation.

[0019] The model update unit is used to iteratively update the model parameters until convergence, so that the cosine similarity of semantically related samples in the shared space exceeds 0.85 (e.g., the similarity between the audio feature of "piano playing" and the text vector of "black grand piano" is >0.9).

[0020] In one embodiment, the present invention provides a cross-modal association recognition system for heterogeneous data understanding, wherein the cross-modal association module includes:

[0021] The spatial vector acquisition unit is used to input the data to be identified (such as user-uploaded images) into the trained model and output a 128-dimensional shared semantic space vector (format example: [0.12,-0.05,0.33,...,0.18]).

[0022] The cross-modal association unit is used to perform tasks based on vector space distance. It calculates the cosine similarity between the query vector and the target library vectors and returns the Top-K results (referring to the top K vectors in the target vector library with the highest cosine similarity to the query vector). The matching task determines whether the similarity between two modal vectors exceeds the first threshold (default threshold = 0.8, which is more lenient than the training threshold of 0.85). The generation task initializes the Transformer decoder (a type of decoder) with conditional vectors to generate target modal content.

[0023] In one embodiment, the present invention provides a cross-modal association recognition system for heterogeneous data understanding, further comprising:

[0024] The iterative update module (following the cross-modal association module) is used to quantify model performance using task-related metrics (e.g., retrieval: Recall@K; matching: Accuracy / F1; generation: BLEU / FID); it analyzes component contributions (e.g., feature extractor, mapping network, loss function) through ablation experiments, and iteratively adjusts model parameters, network structure (e.g., mapping layer dimension), loss function design, or training strategy (learning rate / optimizer / negative sampling) accordingly.

[0025] In one embodiment, the present invention provides a cross-modal association recognition system for heterogeneous data understanding, further comprising:

[0026] The modal conflict detection and resolution module calculates the similarity (e.g., cosine similarity) of each modality feature in the shared semantic space and sets a dynamic second threshold (e.g., adaptively adjusted based on the training data distribution) to determine conflict. If the similarity is lower than the second threshold, it is marked as a conflict state. In the conflict state, a modal confidence weighting mechanism (e.g., based on feature quality or uncertainty score of model output) is introduced to reweight the contribution of each modality, or context-aware resolution is triggered (e.g., arbitration is performed by combining scene metadata or external knowledge base). For related tasks, similarity results with conflict flags are output (e.g., retrieval tasks return a low-confidence Top-K list, matching tasks provide inconsistency judgments, and tasks generate multimodal fusion conditions).

[0027] In one embodiment, the present invention provides a cross-modal association recognition system for heterogeneous data understanding, further comprising:

[0028] The semantic calibration module is used to extract multimodal feature vectors when the modal conflict detection and parsing module outputs low similarity tags, and retrieve typical semantic relationships of the corresponding scene in the common sense base (e.g., "falling down" is usually associated with "groaning" rather than "laughter"). Based on the semantic deviation (e.g., the difference between the actual combination and the probability of matching common sense), it calculates the anti-common sense score. If it exceeds the third threshold, it generates synthetic features that conform to common sense (e.g., replacing conflicting audio with common sense audio features) and inputs them into the downstream cross-modal association module. Otherwise, it retains the original features but adds a risk label.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention collects and processes data by clearly defining the specific application scenarios and heterogeneous modalities involved in cross-modal association recognition, and achieves cross-modal association by sharing distance measures in the semantic space, resulting in high information retrieval efficiency. The designed modal conflict detection and parsing module can effectively identify conflict scenarios, avoiding the failure of the standard cross-modal association process due to the default semantic consistency between modalities in multimodal information conflict scenarios. The designed semantic calibration module further analyzes the conflict scenarios to determine whether they are reasonable. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the first part of a cross-modal association recognition system for understanding heterogeneous data, provided in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the first data acquisition module provided in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of the second data acquisition module provided in an embodiment of the present invention.

[0033] Figure 4 A schematic diagram of the model design module provided in an embodiment of the present invention.

[0034] Figure 5 This is a schematic diagram of a cross-modal association module provided in an embodiment of the present invention.

[0035] Figure 6 This is a schematic diagram of the second part of a cross-modal association recognition system for understanding heterogeneous data, provided in an embodiment of the present invention.

[0036] Figure 7 This is a schematic diagram of the third part of a cross-modal association recognition system for understanding heterogeneous data, provided in an embodiment of the present invention.

[0037] Figure 8 This is a schematic diagram of the fourth part of a cross-modal association recognition system for understanding heterogeneous data, provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script.

[0040] In one embodiment, such as Figure 1 As shown, a cross-modal association recognition system for heterogeneous data understanding includes:

[0041] The first data acquisition module 1 is used to receive the specific application scenario of cross-modal association recognition (such as image-text matching, video question answering or multimodal sentiment analysis) and the heterogeneous modalities involved (such as text, image, audio) based on the specific application scenario, collect data of different modalities, and ensure that the collected data of different modalities are aligned semantically and spatiotemporally (i.e. they describe or perceive the same entity or event), and obtain the first data of different modalities.

[0042] The second data acquisition module 2 is used to independently perform unique cleaning, standardization, and transformation operations (such as text segmentation to remove stop words, image size normalization, and audio resampling and frame division) on the first data of each modality to obtain the second data of each modality. Features are extracted from the second data of each modality (such as text processing by BERT, image processing by ResNet, and audio processing by MFCCs; BERT is a speech model, ResNet is a convolutional neural network architecture, and MFCCs is an audio feature extraction method). The extracted features can capture the core semantic information of the modality.

[0043] Model design module 3 is used to design the multimodal model architecture (including each modal sub-network + shared mapping network), inputting the extracted heterogeneous features into the model; by optimizing specific loss functions (such as contrastive loss and triplet loss), it drives the representation distance of semantically related cross-modal sample pairs (i.e., aligned samples) in the shared semantic space to approach (measured by cosine similarity / Euclidean distance), while pushing away irrelevant sample pairs, thereby learning cross-modal association representations;

[0044] The cross-modal association module 4 is used to map the data to be identified to a shared semantic space using the trained model; it directly implements cross-modal association tasks based on distance metrics (cosine similarity / Euclidean distance) in the space. The cross-modal association tasks include cross-modal retrieval (single-modal query → cross-modal result return), matching (cross-modal sample pair relevance determination), and generation (cross-modal conditional content generation).

[0045] The first data acquisition module 1 ensures targeted data collection by clearly defining the specific application scenario (such as text-to-image matching and video Q&A) and the modalities involved (such as text and images). Its core lies in clarifying the scenario logic through the requirement analysis unit 11 and implementing strict alignment strategies (such as synchronizing video frames and audio with timestamps and binding images and descriptive text with metadata) through the data acquisition and processing unit 12. This ensures that the acquired first data is naturally aligned in semantic and spatiotemporal dimensions, providing a reliable and structured cross-modal data foundation for subsequent feature extraction and model learning, and avoiding content mismatch errors from the source.

[0046] The design of the second data acquisition module 2 stems from the fact that the original forms of heterogeneous modal data (text, image, audio) vary greatly and contain a lot of noise, making them unsuitable for direct model computation. Therefore, the cleaning and standardization unit 21 first performs customized preprocessing for each modality: text needs to be segmented and stop words removed to retain core semantics, images need to be standardized in size and pixels to adapt to convolutional network input, and audio needs to be resampled and framed to capture time-varying features. Subsequently, the format normalization unit 22 converts the processed data into a model-acceptable standard format (such as text to indexed sequence, image to RGB tensor, and audio to Mel spectrogram). Finally, the feature extraction unit 23 uses pre-trained models (BERT, ResNet50, MFCCs) to extract high-level feature vectors containing core semantics (such as text [CLS] vectors, image pooling features, and audio MFCC matrices) from the normalized data. The aim is to transform the original data into computational objects that are rich in semantic information, of moderate dimensionality, and comparable, providing high-quality input for cross-modal association models.

[0047] The core objective of Model Design Module 3 is to learn a model architecture that can map features from heterogeneous modalities to a unified, comparable space. Its design employs a multi-branch structure (text branch, image branch, etc.) to respect and handle the uniqueness of each modality (e.g., the sequential nature of text, the spatial nature of images). Each branch utilizes or fine-tunes a pre-trained backbone (e.g., freezing BERT to extract basic semantics, ResNet to extract visual features) to efficiently capture intra-modal semantics. A key design element is the shared mapping layer (e.g., a two-layer fully connected network), which further projects the features extracted from each branch into a low-dimensional shared semantic space (e.g., 128-dimensional). This space is optimized using contrastive loss (e.g., triplet loss). This loss, through an online hard sample mining strategy, forces the model to narrow the distance (Euclidean distance or cosine similarity) between semantically related samples (e.g., images and their descriptive text) within the shared space, while simultaneously pushing away irrelevant samples. The ultimate goal is to ensure that the similarity of aligned samples exceeds a predetermined threshold (e.g., 0.85), thereby establishing an accurate and robust cross-modal association representation capability within the shared space.

[0048] The cross-modal association module 4 is designed to utilize the trained model and shared semantic space to actually perform cross-modal tasks, with the key being the provision of a unified and efficient interface. The spatial vector acquisition unit 41 inputs the data to be identified (such as images and audio) into the trained model and outputs a unified 128-dimensional shared spatial vector, allowing data from different modalities to be directly compared in the same metric space. The cross-modal association unit 42, based on task requirements and vector distance (such as cosine similarity), implements diverse functions: for retrieval tasks, it calculates the similarity between the query vector and all vectors in the target database and returns the most relevant Top-K results; for matching tasks, it determines relevance by judging whether the similarity between two modal vectors exceeds a preset threshold (such as 0.8, slightly lower than the training threshold to improve generalization); for generation tasks, it uses conditional vectors (such as input image vectors) to initialize the decoder (such as a Transformer) to generate target modal content (such as descriptive text).

[0049] In one embodiment, such as Figure 2 As shown, a cross-modal association recognition system for understanding heterogeneous data includes a first data acquisition module 1 comprising:

[0050] The requirement parsing unit 11 is used to receive the specific application scenario of cross-modal association recognition (such as image and text matching, video question answering or multimodal sentiment analysis) and the heterogeneous modalities involved (such as text, image and audio), and parse the scenario requirements and modal interaction logic.

[0051] The data acquisition and processing unit 12 is used to collect raw data based on scenario requirements and achieve cross-modal semantic / spatiotemporal alignment through timestamp synchronization (aligning video frames and audio segments) or metadata binding (such as associating image IDs with descriptive text), ensuring that different modal data describe the same entity or event and obtaining the first data with structured alignment.

[0052] Different scenarios have fundamentally different requirements for the logic and accuracy of cross-modal data alignment. Image-text matching relies on metadata binding to ensure entity-level semantic association between images and text (e.g., associating product images with descriptive text). Video question-and-answer requires timestamp synchronization to achieve temporal alignment between video clips and question text (e.g., the 10-second frame corresponds to the question "a person is running"). Multimodal sentiment analysis, on the other hand, requires millisecond-level audio-visual synchronization to capture the instantaneous association between micro-expressions and tone of voice. Furthermore, specifying the modality type directly determines the data collection scope and processing flow (e.g., pure image-text scenarios do not require audio frame segmentation, while video question-and-answer requires loading a video parser), avoiding resource waste and preventing the introduction of irrelevant noise. This design ensures that the system adapts to the characteristics of the scenario from the source.

[0053] In one embodiment, such as Figure 3 As shown, a cross-modal association recognition system for understanding heterogeneous data includes a second data acquisition module 2 comprising:

[0054] The cleaning and standardization unit 21 is used to independently perform unique cleaning and standardization on the first data of each modality, perform word segmentation, stop word removal and punctuation mark removal on text data; uniformly scale image data to 224×224 resolution and normalize pixel values ​​to [0,1]; and resample audio data to 16kHz and then perform frame processing (frame length 25ms, frame shift 10ms).

[0055] The format normalization unit 22 is used to convert the cleaned and normalized first data into a normalized format that the model can process, namely the second data. Text data is converted into a word index sequence (mapped to an integer sequence through a vocabulary), image data is converted into a three-channel RGB tensor (dimension 3×224×224), and audio data is converted into a Mel spectrogram (generated by short-time Fourier transform to form a time-frequency matrix).

[0056] The feature extraction unit 23 is used to extract features using a pre-trained model. Text data is input into the index sequence to the BERT model (a pre-trained language model) to obtain a 768-dimensional semantic vector of the [CLS] flag (a special marker); image data is input into the RGB tensor to the ResNet50 (a deep convolutional neural network model) to extract 2048-dimensional features of the final pooling layer; audio data is processed by the MFCCs algorithm (an audio feature extraction method) to calculate a 40-dimensional time-frequency feature matrix for each frame.

[0057] Text data processing needs to eliminate the interference of meaningless words to focus on core semantics (for example, removing "de" and "le" and retaining entity words), so as to meet the semantic extraction requirements of models such as BERT; images are unified in size and pixel-normalized to the range of [0,1], which not only meets the standard input format of the ResNet50 convolutional network, but also eliminates the interference of illumination differences on features; audio is resampled to 16kHz to fully cover the key frequency band of human voice (300-3400Hz), and the framing operation (25ms frame length + 10ms frame shift) simulates the auditory characteristics of the human ear to capture time-varying features (such as plosive / vowel transition), providing standardized input that conforms to acoustic laws for subsequent MFCCs time-frequency analysis.

[0058] In one embodiment, as Figure 4 shows, a cross-modal association recognition system for heterogeneous data understanding, the model design module 3 comprises:

[0059] a model design unit 31, configured to design a multi-branch network architecture, wherein a text branch is composed of a parameter-frozen BERT layer and a trainable BiLSTM layer (bidirectional long short-term memory network), an image branch is composed of a ResNet50 convolutional backbone network and an adaptive pooling layer, and a shared mapping layer is a two-layer fully connected network (with ReLU activation function, outputting 128-dimensional vectors);

[0060] a model training unit 32, configured to adopt an online hard example mining strategy to optimize the triplet loss: in each training, an anchor sample (such as an image) is randomly selected, a positive sample with the same ID (its corresponding text description) and a negative sample with a different ID (other texts) are matched, and back propagation is performed to force the Euclidean distance between the anchor and the positive sample to be at least 0.3 smaller than the distance between the anchor and the negative sample.

[0061] a model updating unit 33, configured to iteratively update model parameters until convergence, so that the cosine similarity of semantically associated samples in the shared space exceeds 0.85 (for example, the similarity between the audio feature of "piano performance" and the text vector of "black grand piano" is greater than 0.9).

[0062] Model design unit 31 adopts a text branch structure of frozen BERT + trainable BiLSTM, aiming to retain the general semantic understanding ability of the pre-trained language model (frozen BERT avoids catastrophic forgetting), while learning task-specific temporal dependencies through trainable BiLSTM (such as negative sentence recognition in sentiment analysis); the image branch reuses ResNet50 convolutional backbone to extract spatial features and adds adaptive pooling layers to dynamically compress feature dimensions to adapt to different input sizes; the shared mapping layer is designed as a two-layer fully connected network, which projects the high-dimensional features of heterogeneous modalities (768-dimensional text / 2048-dimensional image) to a 128-dimensional shared space through nonlinear transformation, preserving discriminative semantics while reducing dimensionality. Model training unit 32 enforces a triplet constraint of "anchor point - positive sample distance < anchor point - negative sample distance - 0.3". The core of this constraint is to set a safety margin of 0.3. This value needs to be sufficient for the model to clearly distinguish similar samples (such as "piano performance" and "violin performance" audio) to prevent mismatches due to small distance differences. At the same time, it avoids excessively large margins (such as >0.5) that would cause optimization difficulties. Experiments have verified that the margin of 0.3 can achieve the optimal balance between discriminability and convergence efficiency, ensuring that semantically related samples are closely clustered in the shared space while irrelevant samples are effectively pushed away.

[0063] In one embodiment, such as Figure 5 As shown, a cross-modal association recognition system for heterogeneous data understanding includes a cross-modal association module 4 comprising:

[0064] The spatial vector acquisition unit 41 is used to input the data to be identified (such as images uploaded by users) into the trained model and output a 128-dimensional shared semantic space vector (format example: [0.12,-0.05,0.33,...,0.18]).

[0065] The cross-modal association unit 42 is used to perform tasks based on vector space distance, calculate the cosine similarity between the query vector and the target library vectors in cross-modal retrieval and return the Top-K results (referring to the top K vectors in the target vector library with the highest cosine similarity to the query vector); the matching task determines whether the similarity between two modal vectors exceeds the first threshold (default threshold = 0.8, which is more lenient than the training threshold of 0.85); the generation task initializes the Transformer decoder (a type of decoder) with conditional vectors to generate target modal content.

[0066] The spatial vector acquisition unit 41 outputs a 128-dimensional shared semantic space vector because lower dimensions (such as 64 dimensions) will result in the loss of semantic information due to insufficient representation ability (such as difficulty in distinguishing the fine-grained differences between "piano playing" and "guitar playing and singing"), while excessively high dimensions (such as 512 dimensions) can retain rich information, but significantly increase the computational cost of cross-modal distance.

[0067] In one embodiment, such as Figure 6 As shown, a cross-modal association recognition system for heterogeneous data understanding also includes:

[0068] Iterative update module 5 (after the cross-modal association module) is used to quantify model performance using task-related metrics (e.g., retrieval: Recall@K; matching: Accuracy / F1; generation: BLEU / FID); analyze component contributions (e.g., feature extractor, mapping network, loss function) through ablation experiments, and iteratively adjust model parameters, network structure (e.g., mapping layer dimension), loss function design, or training strategy (learning rate / optimizer / negative sampling) accordingly.

[0069] Recall@K evaluates cross-modal retrieval performance by calculating the proportion of real-relevant samples in the top K results. For example, Recall@10 = 0.8 means that 8 of the top-10 results are correct. Accuracy / F1 measures the performance of binary classification tasks. Accuracy reflects the overall accuracy, while F1 combines precision and recall, and is particularly suitable for scenarios with imbalanced positive and negative samples, such as security anomaly detection. BLEU / FID quantifies the generation quality. BLEU evaluates the similarity between the generated text and the reference text through n-gram matching (a text similarity evaluation method). FID calculates the Frechet distance between the generated image and the real image in the feature space to evaluate visual fidelity (used to evaluate the distribution distance between the generated image and the real image in the feature space).

[0070] In one embodiment, such as Figure 7 As shown, a cross-modal association recognition system for heterogeneous data understanding also includes:

[0071] Modal conflict detection and parsing module 6 calculates the similarity (e.g., cosine similarity) of each modality feature in the shared semantic space and sets a dynamic second threshold (e.g., adaptively adjusted based on the training data distribution) to determine conflict. If the similarity is lower than the second threshold, it is marked as a conflict state. In the conflict state, a modal confidence weighting mechanism (e.g., based on feature quality or uncertainty score of model output) is introduced to reweight the contribution of each modality, or context-aware parsing is triggered (e.g., arbitration is performed by combining scene metadata or external knowledge base). For related tasks, similarity results with conflict flags are output (e.g., the retrieval task returns a low-confidence Top-K list, the matching task provides inconsistency judgment, and the task fusion multimodal conditions are generated).

[0072] Modal conflict detection and parsing module 6 is inserted after model design module 3 and before cross-modal association module 4. Model design module 3 and cross-modal association module 4 assume natural alignment of multimodal data. However, in practical applications, device malfunctions (such as audio-visual asynchrony), environmental noise (such as audio interference), or semantic ambiguity (such as a "waving" gesture on screen representing both farewell and a call for help) often lead to representational conflicts between modalities. If these conflicts are not processed before entering the association module, misjudgments will occur (such as incorrectly attributing a "running" image and a "cheering" audio message to an attack). Modal conflict detection and parsing module 6 dynamically calculates the similarity of shared space vectors (such as cosine values) and adaptively sets a second threshold based on data distribution (avoiding the sensitivity differences of fixed thresholds in security / medical scenarios). Once a low-similarity conflict is detected, confidence weighting (such as assigning higher weights to high-definition images) or context arbitration (such as calling monitoring location information to determine whether "cheering" is reasonable) is immediately initiated. Finally, association results with conflict flags (such as a low-confidence retrieval list) are output, enabling the system to have conflict perception capabilities. This upgrades the system from passive alignment to proactive anomaly diagnosis, providing a reliable buffer for high-risk decisions.

[0073] In one embodiment, such as Figure 8 As shown, a cross-modal association recognition system for heterogeneous data understanding also includes:

[0074] The semantic calibration module 7 is used to extract multimodal feature vectors when the modal conflict detection and parsing module outputs low similarity tags, and retrieve typical semantic relationships of the corresponding scene in the common sense base (such as "falling down" is usually associated with "groaning" rather than "laughter"); calculate the anti-common sense score based on the semantic deviation (such as the difference between the actual combination and the common sense matching probability). If it exceeds the third threshold, it generates a synthetic feature that conforms to common sense (such as replacing conflict audio with common sense audio features) and inputs it into the downstream cross-modal association module. Otherwise, it retains the original features but adds a risk label.

[0075] The semantic calibration module 7 is located after the modal conflict detection and parsing module 6 and before the cross-modal association module 4. Its core function is to perform semantic rationality arbitration and feature correction on conflict scenarios through an external common sense knowledge base (such as ConceptNet or a large language model).

[0076] While the modal conflict detection and parsing module 6 can identify physical alignment failures (such as audio-visual asynchrony), it cannot explain combinations that are "physically aligned but logically abnormal" (such as a "falling" scene + "laughing" audio). Such counterintuitive events, if left untreated, will lead to high-risk misjudgments (such as misclassifying "an elderly person falling and laughing" as normal behavior in medical monitoring). The semantic calibration module 7 calculates the semantic deviation between the actual multimodal combination and common sense rules by introducing an external common sense knowledge base (such as ConceptNet) (such as "falling + laughing" matching common sense with a probability of only 0.01), generates a counterintuitive score (conflict intensity × probability deviation value), and dynamically synthesizes common sense-compliant features for cases exceeding the third threshold (such as replacing "laughing" with "groaning" MFCC features) and inputs them into the cross-modal association module. At the same time, risk labels are added to cases that do not exceed the threshold.

[0077] For example: Situation A (normal): The visual "person running" + audio "breathing sound" are mapped to the shared space through the model design module 3 (similarity 0.9). The cross-modal association module 4 determines that the match is successful and triggers the "abnormal running alarm".

[0078] Situation B (Physical Conflict): The image is blurry due to camera shake (feature shift), but the "breathing sound" in the audio is clear. The vector similarity output by Model Design Module 3 is only 0.4 (< Dynamic Second Threshold 0.5). Modal Conflict Detection and Parsing Module 6 marks the conflict and starts confidence weighting (audio quality score 0.95 > image quality score 0.6). After weighting, the similarity increases to 0.75. Cross-modal association module 4 outputs matching results with a "low confidence" label (requires manual review) to avoid false alarms of "normal walking".

[0079] Situation C (Counterintuitive Conflict): The video clearly shows "people running", and the audio is "applause and cheers" (as in a funeral scene). The similarity score of 0.3 in the model design module 3 triggers the conflict labeling of the modal conflict detection and parsing module 6, but the confidence weighting cannot explain the rationality. The semantic calibration module 7 searches the common sense base ("running + applause" has a probability of 0.88 in a funeral scene), and the counterintuitive score is 0.2 (< the third threshold of 0.3). Therefore, the original features are retained, but a counterintuitive risk label is added. The cross-modal association module 4 returns the matching results and prompts that the scene is special. Finally, it is manually confirmed as a folk ritual.

[0080] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0081] Those skilled in the art will understand that all or part of the processes in the systems described in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0085] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A cross-modal association recognition system for understanding heterogeneous data, characterized in that, This cross-modal association recognition system for understanding heterogeneous data includes: The first data acquisition module is used to receive the specific application scenario and heterogeneous modalities involved in the cross-modal association recognition input, collect data of different modalities based on the specific application scenario, and ensure that the collected data of different modalities are aligned semantically and spatiotemporally to obtain the first data of different modalities. The second data acquisition module is used to independently perform unique cleaning, standardization and transformation operations on the first data of each modality to obtain the second data of each modality, and extract features from the second data of each modality. The extracted features can capture the core semantic information of the modality. The model design module is used to design multimodal model architectures, inputting extracted heterogeneous features into the model; by optimizing a specific loss function, it drives semantically related cross-modal sample pairs to converge in the representation distance in the shared semantic space, while pushing away irrelevant sample pairs, thereby learning cross-modal association representations; The cross-modal association module is used to map the data to be identified to a shared semantic space using the trained model; it directly implements cross-modal association tasks based on distance metrics in the space, including cross-modal retrieval, matching, and generation. The cross-modal association recognition system for heterogeneous data understanding also includes: The modal conflict detection and parsing module calculates the similarity of each modal feature in the shared semantic space and sets a dynamic second threshold to determine conflict. If the similarity is lower than the second threshold, it is marked as a conflict state. In the conflict state, a modal confidence weighting mechanism is introduced to reweight the contribution of each modality, or trigger context-aware parsing. For related tasks, the similarity results with conflict flags are output. The cross-modal association recognition system for heterogeneous data understanding also includes: The semantic calibration module is used to extract multimodal feature vectors and retrieve typical semantic relationships of the corresponding scene in the common sense base when the modal conflict detection and parsing module outputs low similarity labels. It calculates counterintuitive scores based on semantic deviation. If the score exceeds the third threshold, it generates synthetic features that conform to common sense and inputs them into the downstream cross-modal association module. Otherwise, it retains the original features but adds risk labels.

2. The cross-modal association recognition system for heterogeneous data understanding according to claim 1, characterized in that, The first data acquisition module includes: The requirement parsing unit is used to receive the specific application scenarios and heterogeneous modalities involved in cross-modal association recognition, and to parse the scenario requirements and modal interaction logic. The data acquisition and processing unit is used to collect raw data based on scenario requirements, and achieve cross-modal semantic / spatiotemporal alignment through timestamp synchronization or metadata binding to ensure that different modal data describe the same entity or event, thus obtaining the first data with structured alignment.

3. The cross-modal association recognition system for heterogeneous data understanding according to claim 1, characterized in that, The second data acquisition module includes: The cleaning and standardization unit is used to independently perform unique cleaning and standardization on the first data of each modality. It performs word segmentation, removal of stop words and punctuation marks on text data; it uniformly scales image data to 224×224 resolution and normalizes pixel values ​​to [0,1]; and it resamples audio data to 16kHz and then performs frame processing. The format normalization unit is used to convert the cleaned and normalized first data into a normalized format that the model can process, i.e., the second data. Text data is converted into word index sequences, image data is converted into three-channel RGB tensors, and audio data is converted into Mel spectrograms. The feature extraction unit is used to extract features using a pre-trained model: text data is input into the index sequence and fed into the BERT model to obtain a 768-dimensional semantic vector with the [CLS] flag; image data is input into the RGB tensor and fed into ResNet50 to extract 2048-dimensional features for the final pooling layer; audio data is processed by the MFCCs algorithm to calculate a 40-dimensional time-frequency feature matrix for each frame.

4. The cross-modal association recognition system for heterogeneous data understanding according to claim 1, characterized in that, The model design module includes: The model design unit is used to design multi-branch network architectures. The text branch consists of a BERT layer with frozen parameters and a trainable BiLSTM layer, while the image branch consists of a ResNet50 convolutional backbone network and an adaptive pooling layer. The shared mapping layer is a two-layer fully connected network. The model training unit is used to optimize the triplet loss using an online hard sample mining strategy: each training session randomly selects anchor samples, matches positive samples with the same ID and negative samples with different IDs, and forces the Euclidean distance between the anchor point and the positive sample to be at least 0.3 smaller than the distance between the anchor point and the negative sample through backpropagation. The model update unit is used to iteratively update the model parameters until convergence, so that the cosine similarity of semantically related samples in the shared space exceeds 0.

85.

5. The cross-modal association recognition system for heterogeneous data understanding according to claim 1, characterized in that, The cross-modal association module includes: The spatial vector acquisition unit is used to input the data to be identified into the trained model and output a 128-dimensional shared semantic space vector. The cross-modal association unit is used to perform tasks based on vector space distance. It calculates the cosine similarity between the query vector and the target library vector and returns the Top-K results. The matching task determines whether the similarity between two modal vectors exceeds the first threshold. The generation task initializes the Transformer decoder with conditional vectors to generate target modal content.

6. The cross-modal association recognition system for heterogeneous data understanding according to any one of claims 1 to 5, characterized in that, The cross-modal association recognition system for heterogeneous data understanding also includes: The iterative update module is used to quantify model performance using task-related metrics; it analyzes component contributions through ablation experiments and iteratively adjusts model parameters, network structure, loss function design, or training strategies accordingly.

Images