A Multimodal AI Collaborative Processing System and Method for Translation and Review

Abstract

This application discloses a multimodal AI collaborative processing system and method for translation peer review, relating to the field of data processing technology. The system includes: a data acquisition module for acquiring data to be translated; a preprocessing module for acquiring structured data to be translated; a cross-modal association labeling module for establishing a cross-modal association graph; a cross-modal knowledge enhancement module for acquiring a knowledge-enhanced cross-modal association graph; a context-aware feature extraction module for acquiring text semantic vectors, image feature vectors, speech feature vectors, and text features enhanced by cross-modal association; a dynamic modal weight fusion module for acquiring modal weight configurations; a contradiction pre-screening module for generating a list of potential contradictions; and a cross-modal contradiction detection and classification module for generating misclassification results. This application can solve the information silo problem of traditional single-modal systems and improve the comprehensiveness of peer review.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a multimodal AI collaborative processing system and method for translation and peer review. Background Technology

[0002] Current multimodal AI collaborative processing methods for translation and peer review mainly include rule-based machine translation (RBMT) and statistical machine translation (SMT). RBMT relies on hand-written bilingual dictionaries and grammatical rules, and still maintains high accuracy in specific domains (such as government documents). SMT achieves translation by statistically analyzing word alignment probabilities in a corpus; representative systems such as Moses have achieved excellent results in the WMT competition.

[0003] The existing technology has the following disadvantages:

[0004] (1) Weak ability to handle lexical ambiguity: For example, the semantic differences between the English word "bank" in financial and geographical texts cannot be automatically distinguished;

[0005] (2) Limited by syntactic complexity: The grammatical analysis error rate of long and difficult sentences (such as complex sentences containing more than 3 clauses) exceeds 40%;

[0006] (3) Insufficient coverage of professional fields: The translation accuracy of newly created terms in the biomedical field (such as "CRISPR-Cas9") is less than 65%. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal AI collaborative processing system for translation and peer review to at least solve one of the above-mentioned technical problems.

[0008] One aspect of the present invention provides a multimodal AI collaborative processing system for translation peer review, the multimodal AI collaborative processing system for translation peer review comprising:

[0009] A data acquisition module for translating data is used to acquire data to be translated, which includes text data to be translated, image data to be translated, and audio data to be translated.

[0010] The preprocessing module is used to process the text data, image data, and audio data to be translated into structured data to be translated, respectively. The structured data to be translated includes text structured data, image structured data, and audio structured data.

[0011] A cross-modal association marker module, which is used to build a cross-modal association map based on the structured data to be translated;

[0012] A cross-modal knowledge enhancement module is used to enhance the knowledge of the cross-modal association graph, thereby obtaining a knowledge-enhanced cross-modal association graph;

[0013] A context-aware feature extraction module is used to obtain text semantic vectors, image feature vectors, speech feature vectors, and cross-modal association-enhanced text features based on the knowledge-enhanced cross-modal association graph and text structured data, image structured data, and audio structured data.

[0014] A dynamic modal weight fusion module is used to obtain modal weight configuration based on the text semantic vector, image feature vector, speech feature vector, and knowledge-enhanced cross-modal association graph.

[0015] A contradiction pre-screening module is used to generate a list of potential contradiction points based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, and modal weight configuration.

[0016] A cross-modal contradiction detection and classification module is used to generate misclassification results based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list.

[0017] Optionally, the cross-modal association tagging module includes:

[0018] Generate a multimodal node list based on structured text data, structured image data, and structured audio data;

[0019] Generate an information interaction efficiency matrix based on the multimodal node list;

[0020] Generate initial association clusters based on the information interaction efficiency matrix;

[0021] Generate a cross-modal association map based on the initial association clusters.

[0022] Optionally, the cross-modal knowledge enhancement module includes:

[0023] Obtain a list of knowledge anchor points, which includes multiple sets of knowledge anchor points. Each set of knowledge anchor points includes anchor point ID information, anchor point feature information, and initial verification status.

[0024] A preliminary collaborative verification unit list is generated based on the knowledge anchor list and the cross-modal association graph. The preliminary collaborative verification unit list includes multiple collaborative verification units, and each collaborative verification unit includes component ID information, anchor ID information, and verification efficiency.

[0025] The preliminary list of collaborative verification units is optimized to obtain the optimized list of collaborative verification units.

[0026] Generate adaptive verification configuration based on the optimized list of collaborative verification units;

[0027] Based on the optimized list of collaborative verification units and the adaptive verification configuration, a knowledge-enhanced cross-modal association graph is generated.

[0028] Optionally, optimizing the preliminary collaborative verification unit list to obtain an optimized collaborative verification unit list includes:

[0029] The verification quality entropy of each collaborative verification unit in the collaborative verification unit list is calculated by statistical distribution and information entropy, thereby obtaining the verification quality entropy of each collaborative verification unit. Collaborative verification units with verification quality entropy below the threshold are regarded as abnormal units.

[0030] Adaptive structural reorganization is performed on each anomalous unit to obtain the reorganized unit after reorganization of each anomalous unit;

[0031] Replace each abnormal unit in the collaborative verification unit list with its corresponding recombination unit to obtain the optimized collaborative verification unit list.

[0032] Optionally, the verification quality entropy is calculated using the following formula:

[0033]

[0034] Where K is the number of discretization intervals for the verification efficiency, p k Let q be the probability that the unit efficiency falls in the k-th interval. k Let D be the ideal efficiency distribution and D be the dimension of the input features. Let be the second derivative of efficiency with respect to the d-th dimension feature.

[0035] Optionally, the step of adaptively restructuring each anomalous unit to obtain a recombined unit after restructuring each anomalous unit includes:

[0036] Parameter initialization: Initialize the association weight matrix for each anomalous unit;

[0037] Set initial values ​​for hyperparameters, including: weights for format consistency, efficiency decay, and anchor point diversity.

[0038] Define the reorganization objective loss function, and use the gradient descent method to solve for each anomalous unit until convergence, thereby obtaining the reorganized unit after reorganization of each anomalous unit.

[0039] Optionally, the recombination target loss function is as follows:

[0040]

[0041] Where α is the hyperparameter of the format consistency term; F 术语 F is the eigenvector of the term; 格式 The eigenvectors are in the format ; β is the hyperparameter of the efficiency decay term; F 校验 (τ) represents the verification efficiency at time τ (dynamic efficiency tracking, τ∈[0,T], T is the upper limit of the observation period); The integral over the time domain; γ is the hyperparameter of the anchor diversity term; C is the total number of anchors; θ c θ is the associated angle of the c-th anchor point. max This represents the maximum allowed associated angle.

[0042] Optionally, the context-aware feature extraction module includes:

[0043] A text feature extraction module, which is used to generate text semantic vectors based on the text structured data;

[0044] A visual feature extraction module, which is used to generate an image feature vector based on the image structured data;

[0045] A cross-modal knowledge enhancement module is used to generate cross-modal association-enhanced text features based on the text structured data, image structured data, and the knowledge-enhanced cross-modal association graph.

[0046] An audio feature extraction module is used to generate speech feature vectors based on structured audio data.

[0047] Optionally, the cross-modal contradiction detection and classification module includes:

[0048] XGBoost model acquisition module, which is used to acquire trained XGBoost models;

[0049] The classification module is used to input the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list into the trained XGBoost model to obtain misclassification results.

[0050] This application also provides a multimodal AI collaborative processing method for translation peer review, the multimodal AI collaborative processing method for translation peer review includes:

[0051] Acquire the data to be translated, which includes text data, image data, and audio data to be translated;

[0052] The text data, image data, and audio data to be translated are processed into structured data to be translated, which includes structured text data, structured image data, and structured audio data.

[0053] Establish cross-modal association maps based on the structured data to be translated;

[0054] The cross-modal association graph is augmented with knowledge to obtain a knowledge-enhanced cross-modal association graph.

[0055] Based on the knowledge-enhanced cross-modal association graph and the structured text data, image data, and audio data, obtain text semantic vectors, image feature vectors, speech feature vectors, and cross-modal association-enhanced text features;

[0056] The modality weight configuration is obtained based on the text semantic vector, image feature vector, speech feature vector, and knowledge-enhanced cross-modal association graph;

[0057] A list of potential contradictions is generated based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, and modal weight configuration.

[0058] The misclassification result is generated based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list.

[0059] This application has the following advantages:

[0060] This application integrates text, image, and audio trimodal data. Through multimodal data preprocessing (such as OCR image stream parsing and audio transcription) and context-aware feature extraction (such as text semantic vectors and visual structural features), it achieves a unified representation of cross-modal information, solving the information silo problem of traditional single-modal systems. For example, formula symbols in images can be associated with terms in text, and audio-transcribed text can help verify the colloquial accuracy of translations, improving the comprehensiveness of the review process.

[0061] This application incorporates document type (e.g., "technology" or "medicine") and local context (semantics of neighboring sentences) into the text feature section to avoid terminological ambiguity (e.g., the specific meaning of "neural network" in a medical document). In the visual feature section, it combines image coordinates (positional encoding) and adjacent caption text (e.g., "..."). Figure 1The use of quotation marks ensures that formatting misalignments (such as mismatched images and captions) can be accurately detected. This approach allows feature representations to better reflect the needs of real-world translation scenarios; for example, terminology in scientific papers requires high weighting, while table structures in medical documents require priority validation.

[0062] This application dynamically allocates modal weights based on indicators such as text terminology density and the number of image formula symbols (e.g., the text weight increases to 0.8 when terminology density is high). Combined with enhanced association maps (e.g., the number of images associated with a term), if a term is associated with multiple images, the text weight is automatically increased, prioritizing the detection of terminology mistranslations. This approach can adapt to the review needs of different document types; for example, scientific papers require a focus on verifying terminology consistency, while medical documents require prioritizing the checking of the correspondence between tables and text.

[0063] This application clarifies the correspondence between cross-modal elements by constructing an enhanced association graph (e.g., "neural network" → image ID / formula ID), avoiding missed detections caused by the lack of association between text and images in traditional systems. For example, if "neural network" is mistranslated as "nerve cell" in the translated text, the system can discover through the association graph that the image corresponding to this term (e.g., a neural network structure diagram) has not been correctly referenced, thereby triggering a contradiction detection. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of a multimodal AI collaborative processing system for translation peer review, according to an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0066] like Figure 1 The multimodal AI collaborative processing system for translation and peer review shown includes:

[0067] A data acquisition module for translating data is used to acquire data to be translated, which includes text data to be translated, image data to be translated, and audio data to be translated.

[0068] The preprocessing module is used to process the text data, image data, and audio data to be translated into structured data to be translated, respectively. The structured data to be translated includes text structured data, image structured data, and audio structured data.

[0069] A cross-modal association marker module, which is used to build a cross-modal association map based on the structured data to be translated;

[0070] A cross-modal knowledge enhancement module is used to enhance the knowledge of the cross-modal association graph, thereby obtaining a knowledge-enhanced cross-modal association graph;

[0071] A context-aware feature extraction module is used to obtain text semantic vectors, image feature vectors, speech feature vectors, and cross-modal association-enhanced text features based on the knowledge-enhanced cross-modal association graph and text structured data, image structured data, and audio structured data.

[0072] A dynamic modal weight fusion module is used to obtain modal weight configuration based on the text semantic vector, image feature vector, speech feature vector, and knowledge-enhanced cross-modal association graph.

[0073] A contradiction pre-screening module is used to generate a list of potential contradiction points based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, and modal weight configuration.

[0074] A cross-modal contradiction detection and classification module is used to generate misclassification results based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list.

[0075] In this embodiment, the cross-modal association tagging module includes:

[0076] Generate a multimodal node list based on structured text data, structured image data, and structured audio data;

[0077] Generate an information interaction efficiency matrix based on the multimodal node list;

[0078] Generate initial association clusters based on the information interaction efficiency matrix;

[0079] Generate a cross-modal association map based on the initial association clusters.

[0080] In this embodiment, the text data to be translated, the image data to be translated, and the audio data to be translated are obtained in the following manner:

[0081] For each piece of data to be translated, a unique "data identifier ID" is automatically generated (format: "DT-year-month-day-6-digit random number", e.g., DT-2024-05-20-123456). Simultaneously, metadata is extracted and stored in a metadata database (MySQL 8.0 and above). The metadata extraction rules for different data types are as follows:

[0082] Text data: Extract file name, file size (KB / MB), creation time, modification time, source language (automatically identified by language detection algorithm), number of text pages (DOCX / PDF file), and number of characters (excluding spaces); if it is a PDF file, additionally record whether it contains images (detecting image objects in the page using PDF parsing tools) and whether it is a scanned document (judged by text extractability; if the number of extracted characters is less than 50% of the estimated total number of characters on the page, it is marked as "suspected scanned document").

[0083] Image data: Extract file name, file size, resolution (width × height, e.g., 1920 × 1080), color space (RGB / CMYK), image format, shooting time (if the image contains EXIF ​​information), and image content tags (generated by an image recognition model, such as "formula", "table", "product image", with a tag accuracy of ≥90%); if the image contains text (e.g., poster, screenshot), additionally record the coordinates of the text area (located by the OCR preprocessing module, in the format "top left corner X, top left corner Y, bottom right corner X, bottom right corner Y").

[0084] Audio data: Extract file name, file size, audio format, sampling rate, bit rate, number of channels (mono / stereo), audio duration (minutes:seconds, e.g., 05:30), source language (the speech content is identified through the speech recognition preprocessing module, and then the language is determined, supporting 15 languages ​​including Chinese and English, with a recognition accuracy of ≥95%), and whether it contains silent segments (through audio energy detection, if the energy value of a segment is less than the threshold for more than 1 second, it is marked as a "silent segment").

[0085] In this embodiment, the preprocessing module can perform the following processing:

[0086] For the text data to be translated, structured text data is generated through three steps: cleaning, formatting, and content annotation. The specific operations are as follows:

[0087] First, text data cleaning is performed, automatically identifying and removing invalid information from the text, including meaningless special symbols (such as “□” and other symbols without actual semantic meaning), consecutive spaces and blank lines, as well as repeated page numbers, header and footer labels in the document; for PDF format text, if there is recognition noise generated during the scan-to-text conversion process (such as misrecognition of single characters, such as “f” being misjudged as “t”), it is checked and corrected in combination with contextual semantics to ensure the accuracy of the text content.

[0088] Next, text format standardization was implemented. Text data from different sources (such as DOCX, TXT, and PDF) were uniformly split into a four-level structure of "document-chapter-paragraph-sentence". A unique identifier was assigned to each structural level, for example, the document ID format is "T-Doc-XXX" and the chapter ID is "T-Chap-XXX-XX". At the same time, the hierarchical relationship between each level was clearly recorded, such as a sentence belonging to a specific paragraph. In addition, text metadata was supplemented, including the source file name, the total number of characters in the text (the number after removing invalid characters), the source language identified by the language detection model (supporting mainstream languages ​​such as Chinese, English, Japanese, and Korean, with high recognition accuracy), and the core keywords of each chapter extracted by the TextRank algorithm (about 3 keywords per chapter, used for subsequent semantic association).

[0089] Finally, text content annotation is performed, calling up a terminology database covering vertical fields such as science, medicine, and law. Combining regular expression matching and the BERT entity recognition model, professional terms (such as "CRISPR-Cas9") and proper nouns (such as "Stanford University") in the text are labeled. The annotation information includes the field to which the term belongs, the entity type, and its position in the text. At the same time, the spaCy syntactic analysis tool is used to perform word segmentation, part-of-speech tagging (such as noun "n", verb "v"), and dependency parsing on each sentence to clarify the syntactic relationships such as subject-verb and verb-object, generate a syntactic tree structure, and integrate it into the structured text data to avoid semantic understanding bias caused by syntactic ambiguity during subsequent feature extraction.

[0090] For translated image data (such as PNG, JPG, and BMP formats, including formula diagrams, table diagrams, scene diagrams, etc.), the data is processed in three steps: format unification, quality optimization, and content extraction, to generate structured image data. The specific operations are as follows:

[0091] In the image format standardization stage, all input images are batch converted to PNG lossless compression format, and the color space is uniformly adjusted to RGB (if the original image is in CMYK format, it is processed by a professional color conversion algorithm to ensure that the conversion error is controlled within a small range); the resolution is adjusted according to the image content type, formula graphs and table graphs are uniformly scaled to 1920×1080 resolution (if the original resolution is lower than this value, bilinear interpolation is used to enlarge it; if it is higher, it is scaled down proportionally to maintain the aspect ratio), scene images retain the original resolution, but the resolution parameters need to be recorded; a unique ID is assigned to each image (such as "I-Img-XXX"), and metadata is recorded, including the source file name, format, resolution, color space, image type (formula / table / scene), and the association information with text data (such as the associated text paragraph ID).

[0092] In the image quality optimization stage, a 3×3 Gaussian filtering algorithm is used to remove noise that may exist in the image (such as spots in the scanned image or discolored pixels in the special effects image). During the filtering process, parameters are strictly controlled to avoid over-filtering that could lead to the loss of image details. For images with uneven brightness (such as backlit scenes), a histogram equalization algorithm is used to adjust the image brightness and contrast to make the overall brightness of the image uniform and improve the accuracy of subsequent content recognition. If the image is tilted (such as by angular offset caused by scanning), Hough transform is used to detect the image edge lines, and after calculating the tilt angle, rotation correction is performed to ensure that the image is horizontally aligned.

[0093] In the image content extraction stage, OCR technology (such as TesseractOCR, combined with a domain-trained model) is used to extract text information from the image (such as characters in formulas, data in tables, and annotation text in scene diagrams). The content of the extracted text and its coordinate position in the image are recorded (with the top left corner of the image as the origin, the coordinates of the top left and bottom right corners of the annotated text area). For formula diagrams, a formula recognition model (such as Mathpix) is used to convert the formulas into LaTeX format, which facilitates subsequent association and verification with formula terms in the text. For table diagrams, a table structure detection algorithm is used to identify the row and column boundaries of the table, extract the text content within the table cells, and record the row number, column number, and corresponding text of the cells to generate structured table data. Finally, the image ID, metadata, optimized basic image information, extracted text, and structured content (formula LaTeX format and table data) are integrated to form structured image data.

[0094] For audio data to be translated (such as MP3, WAV, FLAC formats), the process involves three steps: format unification, noise removal, and content transcription and annotation, to generate structured audio data. The specific steps are as follows:

[0095] In the audio format unification stage, all input audio is batch-converted to WAV format (lossless audio format to ensure audio information integrity), with a unified sampling rate of 48kHz, a bit rate of 16bit, and a mono channel count (if the original audio is stereo, it is converted to mono using a channel merging algorithm to avoid channel information interference); each audio segment is assigned a unique ID (e.g., "A-Aud-XXX"), and metadata is recorded, including the source file name, original format, converted format, sampling rate, bit rate, number of channels, total audio duration, and the source language identified by the speech detection model (supporting mainstream languages ​​such as Chinese and English).

[0096] In the audio noise removal stage, a spectral subtraction algorithm is used to first analyze the noise characteristics in the audio (such as background noise and current noise), and then perform noise reduction processing on the audio signal in the frequency band where the noise is concentrated (such as the low frequency band). For silent segments in the audio (determined as silent segments by detecting the audio energy value; when the energy value is lower than a set threshold and the duration exceeds 1 second), the start and end time points of the silent segments are marked, which facilitates the subsequent focused processing of effective audio content and reduces the interference of invalid data on subsequent processes.

[0097] In the audio content transcription and annotation stage, a speech-to-text model (such as the Whisper model, combined with domain-specific fine-tuned data) is invoked to convert effective audio segments into text content. Each sentence in the transcribed text is recorded along with its corresponding audio timestamp (start and end times, accurate to milliseconds). For technical terms in the transcribed text (such as "CT scan" in the medical field and "artificial intelligence" in the scientific field), they are tagged using a domain terminology database, indicating the domain to which the term belongs and its position in the transcribed text. Simultaneously, a speech sentiment analysis model is used to determine the speaker's emotional inclination in the audio (such as neutral, positive, or negative), recording sentiment tags and corresponding audio segment timestamps to provide supplementary information for subsequent cross-modal semantic association. Finally, the audio ID, metadata, denoised audio parameters, transcribed text (including timestamps), terminology annotation information, and sentiment tags are integrated to form structured audio data.

[0098] In this embodiment, the cross-modal association tagging module can perform the following processing:

[0099] For structured text data, text nodes are constructed around the core structure of "sentence-term-entity": each sentence corresponds to a basic text node, and the node information includes the sentence ID (using the unique identifier assigned in the preprocessing stage), the original content of the sentence, and the paragraph ID to which it belongs; professional terms in the text (such as "CRISPR-Cas9" in the medical field and "neural network" in the scientific field) are constructed as separate term nodes, and the node information includes the term ID, the original text of the term, the field to which it belongs, and its position in the text (corresponding to the sentence ID and character offset); proper nouns in the text (such as the institution "Stanford University" and the person "Einstein") are constructed as entity nodes, and the node information includes the entity ID, entity name, entity type (institution / person / location, etc.), and the associated sentence ID.

[0100] For image-structured data, image nodes are constructed around the core concept of "image-image text-image element": each image corresponds to a basic image node, and the node information includes the image ID (using the unique identifier assigned in the preprocessing stage), the image type (formula diagram / table diagram / scene diagram), and the associated text paragraph ID; the text content extracted from the image (such as characters in formulas, data in tables) constructs image text nodes, and the node information includes the image text ID, text content, coordinate position in the image, and the image ID to which it belongs; the key elements in the image (such as the formula structure in the formula diagram, the table rows and columns in the table diagram, and the core objects in the scene diagram) construct image element nodes, and the node information includes the image element ID, element type (formula / table cell / object), element description (such as "formula in LaTeX format", "data in the 2nd row and 3rd column of the table"), and the image ID to which it belongs.

[0101] For structured audio data, audio nodes are constructed around the core structure of "audio segment - transcribed text - audio terminology": each valid audio segment (segment after removing silence) corresponds to a basic audio node, and the node information includes the audio segment ID, the segment start / end timestamp, and the audio ID to which it belongs (using the unique identifier assigned in the preprocessing stage); the transcribed text content is split into sentences, and each sentence constructs an audio transcribed text node, with the node information including the transcribed text ID, text content, corresponding audio segment ID, and timestamp; professional terms in the transcribed text construct audio terminology nodes, with the node information including the audio terminology ID, the original terminology, the field it belongs to, the corresponding transcribed text ID, and its position in the text.

[0102] All nodes in the three modalities of text, image, and audio are aggregated to form a multimodal node list. Each node in the list contains a unique ID, node type (sub-type under the major category of text / image / audio), core information (such as original text, description, timestamp, etc.), and associated structured data identifiers (such as the text paragraph ID, image ID, and audio ID to which it belongs).

[0103] Based on a multimodal node list, an information interaction efficiency matrix is ​​constructed by calculating the information interaction efficiency between nodes of different modalities, thereby quantifying the degree of correlation between nodes.

[0104] First, determine the computational dimensions for information interaction efficiency, and set differentiated computational logic for different modal node combinations (text-image, text-audio, image-audio):

[0105] In calculating the interaction efficiency between text nodes and image nodes, the focus is on combining the semantic matching degree and positional relevance of text content and image information: Semantic matching degree is calculated by the semantic similarity between text nodes (such as sentences and terms) and image nodes (such as image text and image elements). The cosine similarity algorithm is used to convert the relevant text and image content into vectors and then calculate the similarity value (range 0-1, higher values ​​indicate greater semantic similarity); positional relevance is determined by image references mentioned in the text (such as "as..."). Figure 1 As shown in the figure, if the paragraph to which the text node belongs is explicitly associated with an image node, the positional association degree is counted as 1. If there is no explicit reference, the association degree value of 0-0.5 is assigned according to the order of appearance of the text and the image in the document (such as the page number order of the text paragraph and the image). The text-image node interaction efficiency is "semantic matching degree × 0.7 + positional association degree × 0.3", and the result is rounded to 2 decimal places.

[0106] In calculating the interaction efficiency between text nodes and audio nodes, the focus is on the semantic consistency and temporal relevance of the text content and the audio-transcribed text: Semantic consistency is calculated using the semantic similarity (cosine similarity algorithm) between text nodes (sentences, terms) and audio-transcribed text nodes; Temporal relevance is based on audio references mentioned in the text (such as "as described in audio segment 3"). If a text node is explicitly associated with an audio segment node, the temporal relevance is counted as 1. If there is no explicit reference, a relevance value of 0-0.4 is assigned based on the logical order of the text and audio (such as whether the process described in the text is synchronized with the steps explained in the audio); The text-audio node interaction efficiency is "semantic consistency × 0.8 + temporal relevance × 0.2", and the result is rounded to two decimal places.

[0107] In calculating the interaction efficiency between image nodes and audio nodes, the focus is on the semantic association and scene matching degree between image content and audio-transcribed text: Semantic association is calculated by the semantic similarity (cosine similarity algorithm) between image nodes (image text, image element description) and audio-transcribed text nodes; Scene matching degree is based on the consistency between image scene (such as a medical surgical scene image) and audio description scene (such as a surgical procedure explanation). If the scene types are consistent (judged by a scene classification model), the scene matching degree is calculated as 0.6-1 (adjusted according to the level of matching detail), and if they are inconsistent, it is calculated as 0-0.3; The image-audio node interaction efficiency is "semantic association × 0.6 + scene matching degree × 0.4", and the result is rounded to two decimal places.

[0108] Iterate through all pairs of nodes with different modalities in the multimodal node list, calculate the information interaction efficiency value of each pair of nodes, and construct an information interaction efficiency matrix with the node ID as the row and column index. Each element in the matrix is ​​the interaction efficiency value of the corresponding two nodes, thus intuitively presenting the association strength between nodes with different modalities.

[0109] Set information interaction efficiency thresholds, which are dynamically adjusted according to modality differences: the interaction efficiency threshold for text-image nodes is set to 0.6 (because the association between text and images usually requires strong semantic or positional support), the interaction efficiency threshold for text-audio nodes is set to 0.5 (audio transcription may have some errors, so the threshold is appropriately lowered to avoid missing associations), and the interaction efficiency threshold for image-audio nodes is set to 0.45 (the judgment of scene association between images and audio has a certain degree of subjectivity, so the threshold is slightly lower than that for text-related associations). The thresholds can be fine-tuned based on the actual association effects in the future, but this standard should be strictly followed in the initial stage.

[0110] Next, the density clustering algorithm (DBSCAN) is used to cluster the multimodal nodes: taking each node as the core, other modal nodes with interaction efficiency values ​​higher than the corresponding threshold are included in the candidate association set; if the number of nodes in the candidate association set is ≥3 (ensuring that the association cluster has a certain scale and avoiding the isolation of a single node), the core node and the nodes in the candidate set together form an initial association cluster; if the number of nodes in the candidate association set is <3, the node is temporarily marked as a "node to be associated", and further judgment is made on whether to merge it into a similar cluster based on the association situation of other clusters.

[0111] During the clustering process, it is necessary to ensure that each associated cluster contains at least two different modalities of nodes (such as text + image, text + audio, image + audio, or all three modalities) to avoid forming single-modal node clusters (single-modal node clusters cannot reflect the value of cross-modal association); if a single-modal node cluster appears during the clustering process, it should be broken down into independent nodes and re-participated in the allocation of subsequent nodes to be associated.

[0112] Finally, assign a unique cluster ID (format "Cluster-XXX") to each initial associated cluster, record the IDs of all nodes contained in the cluster, the node types, and the maximum interaction efficiency between each node (reflecting the strongest association within the cluster), forming an initial associated cluster list. The list also marks the IDs of "nodes to be associated" and the reasons for not forming a cluster (such as insufficient number of candidate associated nodes).

[0113] The core components of the graph are defined as follows: the graph uses multimodal nodes as "nodes", the information interaction efficiency between nodes as "edges", and the initial association clusters as "subgraph units". Each node retains its complete node information (such as ID, type, core content) in the graph, each edge is labeled with the corresponding interaction efficiency value, and each subgraph unit is labeled with the cluster ID and the modal type composition of the nodes within the cluster (such as "text + image + audio").

[0114] Next, the hierarchical structure of the graph is constructed: the graph is divided into three layers: "core layer - association layer - outer layer". The core layer is the combination of nodes with the highest interaction efficiency in each initial association cluster (for example, if the text term node and the image element node have the highest interaction efficiency in a certain cluster, and they are the core nodes of the text and image modalities respectively, then these two nodes are the core layer of the cluster); the association layer consists of other nodes in the cluster except for the core layer. These nodes are connected to the core layer nodes or other nodes in the association layer through edges. The thickness of the edges is adjusted according to the interaction efficiency value (the higher the efficiency value, the thicker the edge, which intuitively reflects the association strength); the outer layer consists of "nodes to be associated". These nodes are connected to the association cluster with the interaction efficiency closest to the threshold through dashed lines and are marked "association to be confirmed".

[0115] In addition to labeling the interaction efficiency value, the association criteria (such as "semantic matching association", "location reference association", "scene matching association") should also be labeled on the edges connecting nodes to facilitate tracing the association logic during subsequent review processes. For example, the edge between a text sentence node and an image node should be labeled "location reference association (text mentions 'such as...')". Figure 1 The annotation for the connection between the audio transcription text node and the image element node is "scene matching association (both are 'medical surgical scene' + semantic association (similarity 0.52)" and "scene matching association (both are 'medical surgical scene' + semantic association (similarity 0.52)".

[0116] The graph adopts an undirected graph structure (the relationships between nodes are bidirectional, such as the mutual relationship between text nodes and image nodes), and supports viewing the associated nodes, interaction efficiency, and basis of association of any node through the graph browsing tool. At the same time, the graph is stored in a standardized graph data format (such as GraphML) to ensure that the subsequent cross-modal knowledge enhancement module can directly read the graph data and perform knowledge enhancement operations. The stored graph data must include three basic data tables: a node table, an edge table, and a subgraph unit table, which record node information, edge information, and subgraph unit information, respectively. The three are linked by ID to form a complete graph data system.

[0117] In this embodiment, the cross-modal knowledge enhancement module includes:

[0118] Obtain a list of knowledge anchor points, which includes multiple sets of knowledge anchor points. Each set of knowledge anchor points includes anchor point ID information, anchor point feature information, and initial verification status.

[0119] A preliminary collaborative verification unit list is generated based on the knowledge anchor list and the cross-modal association graph. The preliminary collaborative verification unit list includes multiple collaborative verification units, and each collaborative verification unit includes component ID information, anchor ID information, and verification efficiency.

[0120] The preliminary list of collaborative verification units is optimized to obtain the optimized list of collaborative verification units.

[0121] Generate adaptive verification configuration based on the optimized list of collaborative verification units;

[0122] Based on the optimized list of collaborative verification units and the adaptive verification configuration, a knowledge-enhanced cross-modal association graph is generated.

[0123] In this embodiment, the knowledge anchor list contains multiple sets of knowledge anchors, each set encompassing three core elements: anchor ID information, anchor feature information, and initial verification status. The anchor ID information uses a unique encoding format to distinguish different knowledge anchors, ensuring accurate positioning of each anchor in subsequent processing. The anchor feature information is determined based on the domain (e.g., science and technology, medicine, law) involved in the cross-modal association graph. For example, in the medical field, anchor feature information may include standard definitions of professional terms, corresponding image feature descriptions (e.g., typical visual features of specific diseases in medical images), standard pronunciations and semantic explanations of related terms in audio, etc. This feature information is derived from authoritative knowledge bases in the domain (e.g., PubMed in the medical field, IEEE Xplore in the scientific field), ensuring the accuracy and professionalism of the information. The initial verification status is used to mark the initial availability of knowledge anchors, divided into "verified" and "pending verification." Anchors directly obtained from authoritative knowledge bases and manually confirmed are initially verified as "verified," while anchors that only initially meet the requirements but are not yet fully confirmed are initially verified as "pending verification."

[0124] In this embodiment, a preliminary collaborative verification unit list is generated based on the knowledge anchor list and the cross-modal association graph. The preliminary collaborative verification unit list includes multiple collaborative verification units, each of which includes component ID information, anchor ID information, and verification efficiency including:

[0125] The feature information of each node in the cross-modal association graph (such as the terminology content of text nodes, the element description of image nodes, and the transcribed text terminology of audio nodes) is analyzed and matched with the feature information of anchors in the knowledge anchor list to filter out knowledge anchors that are highly related to the features of the graph nodes. Then, using the association clusters in the cross-modal association graph as basic units, at least one set of related knowledge anchors is matched for each association cluster to form a collaborative verification unit. In this embodiment, each collaborative verification unit includes component ID information, anchor ID information, and verification efficiency. The component ID information corresponds to the unique identifier of the association cluster in the cross-modal association graph and is used to associate the collaborative verification unit with the specific association structure in the graph. The anchor ID information is the unique code of the matched knowledge anchor. The verification efficiency is obtained by calculating the feature matching degree between the knowledge anchor and the nodes within the association cluster; the higher the matching degree, the higher the verification efficiency, initially reflecting the verification capability of the collaborative verification unit for the association cluster. All constructed collaborative verification units are summarized to form a preliminary collaborative verification unit list.

[0126] In this embodiment, the preliminary collaborative verification unit list is optimized to obtain an optimized collaborative verification unit list, including:

[0127] The verification quality entropy of each collaborative verification unit in the collaborative verification unit list is calculated by statistical distribution and information entropy, thereby obtaining the verification quality entropy of each collaborative verification unit. Collaborative verification units with verification quality entropy below the threshold are regarded as abnormal units.

[0128] Adaptive structural reorganization is performed on each anomalous unit to obtain the reorganized unit after reorganization of each anomalous unit;

[0129] Replace each abnormal unit in the collaborative verification unit list with its corresponding recombination unit to obtain the optimized collaborative verification unit list.

[0130] In this embodiment, the verification quality entropy is calculated using the following formula:

[0131]

[0132] Where K is the number of discretization intervals for the verification efficiency, p k Let q be the probability that the unit efficiency falls in the k-th interval. k Let D be the ideal efficiency distribution and D be the dimension of the input features. Let be the second derivative of efficiency with respect to the d-th dimension feature.

[0133] Simultaneously capture the efficiency distribution deviation and feature sensitivity of the collaborative verification unit, p k (Unit efficiency interval probability) and q k The difference in the (ideal distribution) reflects the stability of efficiency, while the second derivative term reflects the sensitivity of efficiency to changes in features. This approach avoids the coarseness of traditional "single threshold judgment" and can accurately identify abnormal verification units (such as units with large efficiency fluctuations and sensitivity to features).

[0134] By calculating the H-value of each collaborative verification unit, units with H-values ​​below the threshold are marked as abnormal units (e.g., H-value = 0.35 < threshold 0.5), and subsequently optimized through adaptive structural reorganization. At the same time, the calculation results provide a quantitative basis for the "unit quality labeling" of the knowledge-enhanced association graph (e.g., units with H-values ​​≥ 0.6 are labeled as "high-efficiency verification units"), ensuring that all units entering the cross-modal association construction have stable verification capabilities, avoiding cross-modal association deviations caused by low-quality units, and providing a reliable knowledge support layer for subsequent contradiction detection.

[0135] In this embodiment, the verification quality entropy threshold can be set as needed.

[0136] In this embodiment, the adaptive structural reorganization of each anomalous unit to obtain the reorganized unit after reorganization of each anomalous unit includes:

[0137] Parameter initialization: Initialize the association weight matrix for each anomalous unit;

[0138] Set initial values ​​for hyperparameters, including: weights for format consistency, efficiency decay, and anchor point diversity.

[0139] Define the reorganization objective loss function, and use the gradient descent method to solve for each anomalous unit until convergence, thereby obtaining the reorganized unit after reorganization of each anomalous unit.

[0140] In this embodiment, the recombination target loss function is as follows:

[0141]

[0142] Where α is the hyperparameter of the format consistency term; F 术语 F is the eigenvector of the term; 格式 The eigenvectors are in the format ; β is the hyperparameter of the efficiency decay term; F 校验 (τ) represents the verification efficiency at time τ (dynamic efficiency tracking, τ∈[0,T], T is the upper limit of the observation period); The integral over the time domain; γ is the hyperparameter of the anchor diversity term; C is the total number of anchors; θ c θ is the associated angle of the c-th anchor point. max This represents the maximum allowed associated angle.

[0143] Using the above formula, the unit structure is optimized through multi-objective constraints (such as adjusting the weight of terminology-format association and improving the diversity of anchor points); finally, the H-verification value of the recombined unit is calculated again using the verification quality entropy formula to verify the recombination effect (such as H-verification being improved to 0.62≥0.5), forming a closed loop of "identification-recombination-verification".

[0144] By using the aforementioned verification quality entropy threshold and reorganization target loss function, the following advantages can be achieved: For example, in the peer review of scientific paper translation, a text-image collaborative verification unit was identified as abnormal due to "large efficiency distribution deviation" (H-verification = 0.42). After optimization using the reorganization target loss function (α = 0.7, β = 0.5, γ = 0.3), the unit's terminology-image format matching degree improved from 0.6 to 0.8, and the verification efficiency improved from 0.42 to 0.65. The H-verification was then recalculated as 0.63 ≥ 0.5, thus classifying it as a qualified unit. Through this mechanism, the document explicitly states that the "abnormal unit reorganization qualification rate ≥ 90%" of the cross-modal knowledge enhancement layer provides high-quality unit support for subsequent cross-modal association graph construction, avoids cross-modal association failures caused by unit quality issues, and solidifies the core foundation of the system's cross-modal collaboration.

[0145] In this embodiment, generating the adaptive verification configuration based on the optimized collaborative verification unit list includes:

[0146] The distribution of verification efficiency of each unit in the optimized collaborative verification unit list was statistically analyzed to determine the characteristic differences between high-efficiency verification units (verification efficiency higher than 1.2 times the average) and conventional verification units.

[0147] For different types of association clusters (such as text-image association clusters, text-audio association clusters, image-audio association clusters, and three-modal hybrid association clusters), the anchor point type preference of the matched collaborative verification units is analyzed (e.g., text-image association clusters are more suitable for knowledge anchor points containing visual feature descriptions). Finally, based on the above analysis results, corresponding verification parameters are configured for different types of association clusters (which can be configured as needed), including verification frequency (the verification frequency of association clusters corresponding to high-efficiency verification units can be appropriately reduced, while the verification frequency of association clusters corresponding to regular verification units needs to be increased), verification priority (the verification priority of core association clusters (such as association clusters containing key terms) is set to high, and secondary association clusters are set to medium or low), and anchor point invocation strategy (prioritizing the invocation of knowledge anchor points in the "verified" state, with "pending verification" anchor points only used as supplements). These parameters together constitute an adaptive verification configuration, ensuring that the verification work can be flexibly adjusted according to the actual situation of the association clusters, thereby improving verification efficiency and accuracy.

[0148] In this embodiment, generating a knowledge-enhanced cross-modal association graph based on the optimized collaborative verification unit list and adaptive verification configuration includes:

[0149] Each collaborative verification unit in the optimized collaborative verification unit list is bound to the corresponding association cluster in the cross-modal association graph. A corresponding collaborative verification unit identifier and verification efficiency information are added to each association cluster in the graph to establish a direct association channel between the association cluster and the knowledge anchor.

[0150] Based on the adaptive verification configuration, the verification parameters (verification frequency, verification priority, anchor point calling strategy) are labeled for different types of association clusters in the graph, enabling the graph to have dynamic verification capabilities. At the same time, the anchor feature information of the "verified" status in the knowledge anchor list is supplemented into the attributes of the corresponding nodes in the graph. For example, authoritative definitions are added to text term nodes, and standard visual feature descriptions are added to image element nodes, improving the richness and authority of node information.

[0151] The graph structure is optimized by removing invalid associations discovered through knowledge enhancement (such as association edges that do not match the knowledge anchors at all) and strengthening effective associations (such as association edges corresponding to collaborative verification units with high verification efficiency). The result is a knowledge-enhanced cross-modal association graph that not only retains the original cross-modal associations but also incorporates authoritative knowledge support and dynamic verification configuration, which can better support subsequent context-aware feature extraction and cross-modal contradiction detection.

[0152] In this embodiment, the context-aware feature extraction module includes:

[0153] A text feature extraction module, which is used to generate text semantic vectors based on the text structured data;

[0154] A visual feature extraction module, which is used to generate an image feature vector based on the image structured data;

[0155] A cross-modal knowledge enhancement module is used to generate cross-modal association-enhanced text features based on the text structured data, image structured data, and the knowledge-enhanced cross-modal association graph.

[0156] An audio feature extraction module is used to generate speech feature vectors based on structured audio data.

[0157] In this embodiment, the text feature extraction module includes:

[0158] Extract the hierarchical structure of "document-chapter-paragraph-sentence" and the corresponding text content from the structured text data, with a focus on sentence-level text (using the sentence ID identifier assigned in the preprocessing stage);

[0159] By combining the knowledge-enhanced cross-modal association graph, information about other modal nodes associated with each sentence node (such as descriptions of associated image element nodes and content of audio-to-text nodes) is obtained. This association information is then used as supplementary contextual information for the sentence and integrated into the original sentence content to form an extended text consisting of "original sentence + cross-modal contextual supplementation".

[0160] The extended text is encoded using a semantic coding model. During the encoding process, the model captures the semantic information of the original sentence and the related semantic information of the cross-modal context. Finally, it outputs a fixed-dimensional text semantic vector. This vector not only contains the semantic features of the text itself, but also incorporates the contextual semantics of cross-modal association, which can more comprehensively reflect the semantic connotation of the text in multimodal scenarios.

[0161] In this embodiment, the visual feature extraction module includes:

[0162] Extract optimized basic image information (such as resolution and color space) and image element descriptions (such as formula LaTeX format, table row and column data, and scene object descriptions) from image structured data;

[0163] By combining the knowledge-enhanced cross-modal association graph, the text node information associated with the image node (such as term definitions and sentence semantics in the associated text paragraphs) is obtained, and this text association information is transformed into a supplementary description of the visual features (for example, if the image is a medical formula diagram, the description in the associated text "this formula is used to calculate the drug metabolism rate" will serve as a semantic supplement to the visual features of the image).

[0164] A visual feature extraction model is used to process images. The model first extracts the low-level visual features of the image (such as pixel distribution, edge texture, and color histogram), and then combines them with cross-modal associated textual supplementary descriptions to semantically enhance the low-level visual features. Finally, an image feature vector is generated. This vector contains both the visual structural features of the image and carries cross-modal associated semantic information, which can support subsequent collaborative analysis with text and audio features.

[0165] In this embodiment, the cross-modal knowledge enhancement module includes:

[0166] Extract text terminology node information (such as the original text of the term, its domain, and its position in the text) from the structured text data; combine it with the knowledge-enhanced cross-modal association graph to filter out image element nodes associated with the term node (such as the formula diagram or schematic diagram corresponding to the term), and extract the visual feature descriptions of these image element nodes (such as formula structure features, image color and shape features).

[0167] The feature information of terms (such as semantics and domain attributes) is fused with the visual feature descriptions of associated image elements. During the fusion process, the association features between terms and image elements are strengthened (for example, the term "neural network" and the associated neural network structure diagram image elements will be highlighted through association weight calculation to emphasize their common features in "hierarchical structure" and "node connection"). Finally, the fused features are integrated with the semantic features of the original text terms to generate cross-modal association-enhanced text features. These features not only retain the semantic attributes of the text terms themselves but also incorporate the visual association features of the associated images, which can significantly improve the accuracy of text features in cross-modal contradiction detection.

[0168] In this embodiment, the audio feature extraction module includes:

[0169] Extract the basic audio parameters (such as sampling rate and bit rate), timestamps of effective audio segments, and corresponding transcribed text from the structured audio data;

[0170] By combining the knowledge-enhanced cross-modal association graph, we can obtain the text nodes (such as terms and entities in the associated text sentences) and image nodes (such as scene descriptions in the associated images) associated with the audio segment nodes, and use this association information as a contextual semantic reference for the audio segment.

[0171] A speech feature extraction model is used to process effective audio segments. The model first extracts the acoustic features of the audio (such as Mel frequency cepstral coefficients, fundamental frequency, and audio energy), and then combines the semantic information of the transcribed text with the contextual semantic references associated with cross-modal contexts to perform semantic association optimization on the acoustic features. Finally, a speech feature vector is output. This vector contains both the acoustic characteristics of the audio and the semantic features associated with cross-modal contexts, which can more accurately represent the characteristics of the audio in multimodal review scenarios.

[0172] In this embodiment, the dynamic modal weight fusion module includes:

[0173] The input vectors are standardized. For example, if the original text semantic vectors are of non-fixed dimensions (e.g., different sentence lengths correspond to different dimensions), mean pooling is used to compress them to a fixed dimension to eliminate length differences. The image feature vectors retain the vectors output by ResNet and are reduced to the [0,1] interval by L2 normalization to avoid the influence of numerical range differences on the association calculation. The speech feature vectors extracted by MFCC are subjected to frame-level mean fusion to merge the speech features of multiple frames into a single vector, ensuring alignment with the "single sample - single vector" format of text and image vectors.

[0174] From the knowledge-enhanced cross-modal association graph, core information directly related to weight calculation is derived to form an "association information table":

[0175] Modal node association records: Extract the association relationships between text nodes (e.g., T-Sen-XXX) and image nodes (e.g., I-Img-XXX), text nodes and audio nodes (e.g., A-Seg-XXX), and image nodes and audio nodes, and label the interaction efficiency value of each pair of associated nodes (using the results calculated by the cross-modal association labeling module, such as text-image interaction efficiency 0.78, text-audio interaction efficiency 0.65);

[0176] Cooperative verification unit information: Extract the "verification efficiency" of the cooperative verification unit corresponding to each modal node (e.g., the verification unit efficiency associated with the text node is 0.82, and the verification unit efficiency associated with the image node is 0.75);

[0177] Modal node attributes: extract the "term density" of text nodes (the proportion of technical terms in the corresponding text, such as 18%), the "type label" of image nodes (formula diagram / table diagram / scene diagram), and the "transcription accuracy" of audio nodes (the semantic matching degree between the transcribed text and the associated text, such as 92%).

[0178] Based on the interaction efficiency values ​​in the "Association Information Table", the association strength of the three modal combinations of text-image, text-audio, and image-audio is quantified to provide a coefficient basis for weight adjustment, as detailed below:

[0179] Set an interaction efficiency threshold to divide the association strength into three levels:

[0180] Strong association: Interaction efficiency ≥ 0.7 (e.g., text-image interaction efficiency 0.78 is considered a strong association);

[0181] Medium correlation: 0.4 ≤ interaction efficiency < 0.7 (e.g., text-to-audio interaction efficiency of 0.65 is considered medium correlation);

[0182] Weak association: Interaction efficiency < 0.4 (e.g., image-audio interaction efficiency of 0.35 is considered weak association).

[0183] Assign corresponding adjustment coefficients to different correlation strengths for subsequent weight amplification or constraint:

[0184] Strong correlation coefficient: 1.2 (the weight of closely correlated modes needs to be increased);

[0185] Medium correlation coefficient: 1.0 (modes in medium correlation maintain basic weights);

[0186] Weak correlation coefficient: 0.8 (The weight of weakly correlated modes should be appropriately reduced to avoid interference from invalid information).

[0187] Construct a cross-modal association matrix. For example, using "text, image, and audio" as rows and columns, construct a 3×3 association strength coefficient matrix, as shown in the example below:

[0188] Text-Image: 1.2 (Strongly related); Text-Audio: 1.0 (Mediumly related); Text-Text: 1.0 (Self-related, no adjustment);

[0189] Image-Text: 1.2 (strong correlation); Image-Audio: 0.8 (weak correlation); Image-Image: 1.0;

[0190] Audio-Text: 1.0 (Medium correlation); Audio-Image: 0.8 (Weak correlation); Audio-Audio: 1.0.

[0191] By combining modal vector features and association graph node attributes, the "basic weights" (without considering initial weights for cross-modal associations) of text, image, and audio are calculated separately. The calculation logic for each modality corresponds to the multimodal features and association information mentioned in the document, as follows:

[0192] The text's base weight is obtained by summing two parts: "term density contribution" and "association breadth bonus," with a value range of [0.3, 0.5].

[0193] Term density contribution: The "term density" of text nodes is mapped to a base score, such as 0.4 for term density ≥15%, 0.35 for 10%-14%, and 0.3 for less than 10%.

[0194] Association breadth bonus: Count the number of other modal nodes (total number of image + audio nodes) associated with a text node. Add 0.1 for ≥3 associated nodes, add 0.05 for 1-2 associated nodes, and add 0 for 0 associated nodes.

[0195] Calculation formula: W text_base =Term density contribution + association breadth bonus.

[0196] For example, if the term density is 18% (contribution 0.4), and it is associated with 2 image nodes + 1 audio node (additional score 0.1), then W text_base =0.4 + 0.1 = 0.5.

[0197] In this embodiment, the basic image weight is obtained by summing the "visual information density" and the "verification efficiency bonus", with a value range of [0.2, 0.4]:

[0198] Visual information density: Assigned based on the "type label" of the image node, 0.35 for formula graphs / table graphs (carrying precise information) and 0.25 for scene graphs (carrying scene information);

[0199] Additional score for verification efficiency: The "verification efficiency" of the collaborative verification unit associated with the image node is taken. If the efficiency is ≥0.8, add 0.05; if it is 0.6-0.79, add 0.03; if it is less than 0.6, add 0.

[0200] Calculation formula: W img_base = Visual information density + verification efficiency bonus.

[0201] Example: If the image type is a formulaic image (density 0.35), the verification efficiency is 0.82 (additional score 0.05), then W img_base =0.35 + 0.05 = 0.4.

[0202] In this embodiment, the basic audio weight is obtained by summing "acoustic semantic matching degree" and "emotional tag validity", with a value range of [0.1, 0.3]:

[0203] Acoustic semantic matching degree: The "transcription accuracy" of audio nodes is mapped to a basic score, with an accuracy of ≥90% corresponding to 0.25, 80%-89% corresponding to 0.2, and less than 80% corresponding to 0.15;

[0204] Sentiment tag validity: If the sentiment tag of the audio node (such as "neutral" or "professional explanation") is relevant to the translation and review scenario (such as the "serious" sentiment tag being valid in legal documents), add 0.05; otherwise, add 0.

[0205] Calculation formula: W audio_base = Acoustic semantic matching degree + sentiment tag effectiveness.

[0206] For example, if the transcription accuracy is 92% (matching score 0.25) and the sentiment tag "professional explanation" is effective (bonus score 0.05), then W audio_base =0.25 + 0.05 = 0.3.

[0207] By combining the basic weights of each unimodal mode with the correlation strength coefficient matrix, the weights of each mode are adjusted to ensure that closely correlated modes receive a weight boost.

[0208] Text weight adjustment: W text_adjust =W text_base ×(text-image correlation coefficient + text-audio correlation coefficient) / 2 (take the average of the two correlation coefficients to avoid excessive influence from a single correlation);

[0209] Image weight adjustment: W img_adjust =W img_base ×(image-text correlation coefficient + image-audio correlation coefficient) / 2;

[0210] Audio weight adjustment: W audio_adjust =W audio_base ×(audio-text correlation coefficient + audio-image correlation coefficient) / 2.

[0211] Since the adjusted total weight may not be equal to 1, it is necessary to compress the weight to the [0,1] interval through "L1 normalization" to ensure that the total weight of the three modalities is 1.

[0212] In this embodiment, the modal weight configuration includes the following:

[0213] Modal weight values, such as text weight (e.g., 0.451), image weight (e.g., 0.328), and audio weight (e.g., 0.221);

[0214] The base weights for each modality (e.g., text base weight 0.5), the association strength coefficient (e.g., text-image coefficient 1.2), and the pre-normalization adjustment values ​​(e.g., text adjusted value 0.55);

[0215] Applicable scenario annotation: Based on the feature annotation configuration of the association graph, the applicable review scenarios are configured (such as "scientific paper - formula-intensive document", "medical report - mixed text and image document").

[0216] In this embodiment, the contradiction pre-screening module includes:

[0217] Based on the priority determined by the modal weight configuration, the feature consistency within each modality of text, image, and audio is analyzed to identify feature points that may contain contradictions within each modality.

[0218] Text Modal Internal Difference Analysis: Based on text semantic vectors and text structured data, the semantic consistency of the same terms within the same document is compared. Specifically, recurring technical terms (such as "CRISPR-Cas9" and "neural network") are extracted from the text. The similarity of the text semantic vectors corresponding to the terms at different positions is calculated (using the cosine similarity algorithm). If the similarity is <0.7 (indicating a semantic deviation in the term) and the verification efficiency of the co-validation unit associated with the term is <0.6 (indicating a lack of authoritative knowledge support for the term), then the different occurrence positions of the term are marked as potential contradiction points, and the contradiction positions (corresponding sentence IDs and character offsets) and semantic difference values ​​are recorded.

[0219] Image modality internal difference analysis: Combining image feature vectors and image structured data, the visual feature consistency of images of the same type (such as formula diagrams and table diagrams) is analyzed. Specifically, images of the same type in the image structured data are selected, and the similarity (cosine similarity) of their image feature vectors is calculated. If the similarity is <0.6 (indicating significant differences in visual features), and there are differences in the text descriptions associated with the images (e.g., the text description associated with one formula diagram is "drug metabolism formula", while the text description associated with another similar formula diagram is "cell division formula"), then these two types of images are marked as potential contradictions, and the image ID, visual feature difference value, and differences in associated text descriptions are recorded.

[0220] Audio modality internal difference analysis: Based on speech feature vectors and audio structured data, the semantic consistency of the same term in the audio-transcribed text is verified. Specifically, repeated terms in the audio-transcribed text are extracted, and the similarity (cosine similarity) of the corresponding speech feature vectors is calculated. If the similarity is <0.65 (indicating a deviation in speech semantics) and the semantic matching degree between the transcribed text and the associated text is <0.8 (indicating doubts about the accuracy of transcription), then different audio segments of the term are marked as potential contradiction points, and the audio segment ID, timestamp, and basis for semantic deviation are recorded.

[0221] Based on the knowledge-enhanced cross-modal association graph and modal weight configuration, the consistency of feature associations among different modalities of text, image, and audio is verified, and potential contradictions at the cross-modal level are identified.

[0222] Text-image association consistency verification: Taking the text modality as the core (based on weight priority), the consistency of association between the text semantic vector and the associated image feature vector is compared. Specifically, strong association combinations of text nodes and image nodes (interaction efficiency ≥ 0.6) are selected from the association graph. The semantic description of the term in the text (such as "circular cell structure diagram") is compared with the description of the image element (such as "square cell structure diagram" recorded in the image structured data). If there is a significant conflict between the descriptions and the similarity between the two feature vectors is < 0.65, the text-image association combination is marked as a potential contradiction point, and the text sentence ID, image ID, conflicting description content, and feature similarity value are recorded.

[0223] Text-Audio Association Consistency Verification: Based on the text modality, the semantic consistency between the text semantic vector and the associated speech feature vector is verified. Specifically, the association combination of text nodes and audio nodes in the association graph is extracted (interaction efficiency ≥ 0.55). The semantic content of the text sentence is compared with the audio-transcribed text content. If there are differences in key information (such as numerical values, terminology, logical relationships) and the similarity between the text semantic vector and the speech feature vector is < 0.7, the text-audio association combination is marked as a potential contradiction point, and the text sentence ID, audio segment ID, timestamp, and key information difference points are recorded.

[0224] Image-audio association consistency verification: Combining modal weight configuration (the sum of image and audio weights), the scene consistency between image feature vectors and associated speech feature vectors is analyzed. Specifically, image node and audio node association combinations (scene matching degree ≥ 0.45) are selected from the association graph. The scene description of the image (e.g., "medical surgical scene diagram") is compared with the scene description in the audio transcription text (e.g., "explanation of chemical experiment operation"). If the scene types are inconsistent and the similarity between the image feature vector and the speech feature vector is < 0.6, the image-audio association combination is marked as a potential contradiction point, and the image ID, audio segment ID, and scene conflict content are recorded.

[0225] All potential conflicting points identified within and across single modes are prioritized based on modal weight configuration and correlation graph information, ultimately generating a structured list of potential conflicting points.

[0226] Conflict Priority Assessment: The basic priority of conflict points is determined based on modal weights. For example, text-related conflict points (single-modal or cross-modal) have the highest basic priority (weight ratio refers to text weight), image-related conflict points are second (refer to image weight), and audio-related conflict points have the lowest basic priority (refer to audio weight). Then, the verification efficiency value of the corresponding collaborative verification unit is combined. The lower the verification efficiency (indicating a higher conflict risk), the greater the priority increase. For example, the basic priority of a text-image cross-modal conflict point is 0.45. If the verification efficiency is 0.4 (below the threshold of 0.6), the priority is increased to 0.6.

[0227] Supplementing information on points of conflict: Complete association information is provided for each potential point of conflict, including the type of conflict (single-modal text conflict, cross-modal text-image conflict, etc.), the IDs of the modal nodes involved (text sentence ID, image ID, audio segment ID), the criteria for conflict determination (feature similarity value, description of conflict content, verification efficiency value), and the priority level (high / medium / low, divided according to the priority evaluation results, such as priority ≥0.6 is high, 0.4-0.59 is medium, and less than 0.4 is low).

[0228] Potential conflict point list integration: All potential conflict points are sorted from high to low priority, with high-priority conflict points (such as core terminology text-image description conflicts) at the top and low-priority conflict points (such as semantic deviations of non-critical audio segments) at the bottom; the list clearly defines the complete information of each conflict point, ensuring that the subsequent cross-modal conflict detection and classification modules can directly call it to quickly locate high-risk conflict points for in-depth detection.

[0229] In this embodiment, the cross-modal contradiction detection and classification module includes:

[0230] XGBoost model acquisition module, which is used to acquire trained XGBoost models;

[0231] The classification module is used to input the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list into the trained XGBoost model to obtain misclassification results.

[0232] In this embodiment, the "multimodal fusion feature vector" is concatenated with the "key information of the knowledge-enhanced association graph" and the "modal weight coefficient". Specifically, the numerical information such as the verification efficiency value and knowledge anchor matching degree (such as the semantic similarity between terms and anchor definitions) in the association graph are first converted into feature values ​​(within the range of [0,1]). Then, the text, image and audio weight values ​​in the modal weight configuration are used as modal importance features, and finally a complete input feature set of "multimodal fusion feature + knowledge association feature + weight feature" is formed to adapt to the numerical input requirements of the XGBoost model.

[0233] In this embodiment, the output of the XGBoost model includes a contradiction determination conclusion (with a value range of [0,1], representing the probability that the sample output by the model belongs to a "true contradiction"), multi-dimensional classification labels (e.g., "single-modal contradiction" (such as "single-modal contradiction-text", "single-modal contradiction-image", "single-modal contradiction-audio"), "bimodal contradiction" (such as "bimodal contradiction-text-image", "bimodal contradiction-text-audio", "bimodal contradiction-image-audio"), "multimodal contradiction" (i.e., "multimodal contradiction-text-image-audio"), which fully covers the modal association scenarios mentioned in the document), auxiliary information (e.g., "core feature similarity" (e.g., the similarity between the text semantic vector and the image feature vector is 0.62), "knowledge anchor support" (e.g., "the matching degree between the term and the verified knowledge anchor is 0.65, and there is a deviation"), and "co-validation results" (e.g., "co-validation unit efficiency is 0.6, which is lower than the threshold of 0.7")).

[0234] This application has the following advantages:

[0235] By integrating text, image, and audio data across the entire process of "data input - data preprocessing - cross-modal association - cross-modal knowledge enhancement - cross-modal conflict detection layer," this system addresses the information isolation problem inherent in traditional single-modal systems (such as translation and peer review tools that only process text). For example, after the preprocessing layer transforms text, images, and audio into structured data, the cross-modal association tagging layer constructs a cross-modal association graph based on standardized data. This allows technical terms in the text to be associated with formulas in the images and speech descriptions in the audio (e.g., the term "CRISPR-Cas9" is simultaneously associated with both the image formula diagram and the audio explanation segment). This multimodal collaborative mechanism covers the entire scenario of "text + image + audio" in translation and peer review, avoiding the omissions caused by traditional systems that only process a single modality (such as ignoring descriptive conflicts between images and text, or semantic discrepancies between audio and text), thus improving the comprehensiveness of multimodal information coverage in peer review.

[0236] The data preprocessing layer transforms unstructured data into standardized structured data, providing a precise foundation for cross-modal contradiction detection. Simultaneously, an authoritative knowledge anchor is introduced through a cross-modal knowledge enhancement layer, further strengthening the accuracy of contradiction determination. For example, in the cross-modal contradiction detection layer, the system uses text semantic vectors and image feature vectors from the structured data, combined with knowledge anchors (such as the standard definition of terms in the "verified" state), to perform a secondary verification of potential contradictions. If the semantics of the text term "neural network" do not match the anchor definition and conflict with the visual description of the associated image, the system can accurately determine it as a high-priority contradiction point of "terminology mistranslation + cross-modal description conflict." Compared to traditional systems that rely solely on single-modal features or human experience, this dual support of "structured data + knowledge anchors" significantly reduces the contradiction misjudgment rate (document implicit misjudgment rate ≤ 10%) and improves the accuracy of contradiction detection.

[0237] In the dynamic modal weight fusion layer, weights for the three modalities are dynamically allocated based on indicators such as text terminology density, image information value, and audio semantic importance (e.g., the weight of Chinese fonts in scientific papers is increased to 0.8, while the weight of non-critical audio is reduced to 0.2). In the contradiction pre-screening layer and the cross-modal contradiction detection layer, potential contradictions are prioritized based on weights, with high-priority contradictions (such as text-image conflicts of core terms) being detected first, and low-priority contradictions (such as non-critical audio deviations) being processed later. This design allows review resources (such as manual review and model computing power) to be concentrated on high-value contradictions, avoiding the inefficiency waste caused by the traditional system's "full detection and indiscriminate processing." For example, in a 15-second medical report review scenario, the system can prioritize high-priority contradictions such as "mistranslation of core terms" and "conflict between text and image descriptions," improving review efficiency by more than 30% compared to traditional systems, thus meeting the core need of "improving review efficiency" in documents.

[0238] In the cross-modal contradiction detection and classification layer, misclassification results are broken down into three parts: "contradiction judgment conclusion, multi-dimensional classification labels, and auxiliary information." The multi-dimensional classification labels cover "modal dimension classification" (unimodal / bimodal / multimodal) and "error nature classification" (terminological mistranslation / descriptive conflict / format inconsistency, etc.). Auxiliary information includes the basis for contradiction judgment (feature similarity, knowledge anchor matching results) and the IDs of the involved modal nodes (text sentence ID, image ID, etc.). For example, a misclassification result might be labeled "bimodal contradiction - text - image + terminological mistranslation," while also providing auxiliary information such as "similarity between text semantic vector and image feature vector 0.61" and "knowledge anchor verification conflict." Reviewers can directly use this information to quickly locate the error and understand its cause without having to revisit previous processes. This structured classification provides clear guidance for manual review and automated correction, supporting accurate review decisions.

[0239] This application also provides a multimodal AI collaborative processing method for translation peer review, the multimodal AI collaborative processing method for translation peer review includes:

[0240] Acquire the data to be translated, which includes text data, image data, and audio data to be translated;

[0241] The text data, image data, and audio data to be translated are processed into structured data to be translated, which includes structured text data, structured image data, and structured audio data.

[0242] Establish cross-modal association maps based on the structured data to be translated;

[0243] The cross-modal association graph is augmented with knowledge to obtain a knowledge-enhanced cross-modal association graph.

[0244] Based on the knowledge-enhanced cross-modal association graph and the structured text data, image data, and audio data, obtain text semantic vectors, image feature vectors, speech feature vectors, and cross-modal association-enhanced text features;

[0245] The modality weight configuration is obtained based on the text semantic vector, image feature vector, speech feature vector, and knowledge-enhanced cross-modal association graph;

[0246] A list of potential contradictions is generated based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, and modal weight configuration.

[0247] The misclassification result is generated based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list.

[0248] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A multimodal AI collaborative processing system for translation and peer review, characterized in that, The multimodal AI collaborative processing system for translation and peer review includes: A data acquisition module for translating data is used to acquire data to be translated, which includes text data to be translated, image data to be translated, and audio data to be translated. The preprocessing module is used to process the text data, image data, and audio data to be translated into structured data to be translated, respectively. The structured data to be translated includes text structured data, image structured data, and audio structured data. A cross-modal association marker module, wherein the cross-modal association marker module is used to establish a cross-modal association map based on the structured data to be translated; the establishment of the cross-modal association map based on the structured data to be translated includes: All nodes of the three modalities of text, image, and audio are aggregated to form a multimodal node list. Each node in the list contains a unique ID, node type, core information, and associated structured data identifier. Based on a multimodal node list, an information interaction efficiency matrix is ​​constructed by calculating the information interaction efficiency between nodes of different modalities. Density clustering algorithm is used to cluster multimodal nodes: with each node as the core, other modal nodes with interaction efficiency values ​​higher than the corresponding threshold are included in the candidate association set; if the number of nodes in the candidate association set is ≥3, the core node and the nodes in the candidate set together form an initial association cluster; if the number of nodes in the candidate association set is <3, the node is marked as a node to be associated. Each initial associated cluster is assigned a unique cluster ID, and records the IDs of all nodes contained in the cluster, the node types, and the maximum interaction efficiency between each node, forming an initial associated cluster list; Determine the core components of the graph: The graph uses multimodal nodes as nodes, the information interaction efficiency between nodes as edges, and the initial association clusters as subgraph units; The hierarchical structure of the graph is constructed as follows: The graph consists of three layers: core layer, association layer, and outer layer. The core layer is the combination of nodes with the highest interaction efficiency in each initial association cluster, the association layer consists of other nodes in the cluster except for the core layer, and the outer layer consists of nodes to be associated. A cross-modal knowledge enhancement module, used to enhance the cross-modal association graph with knowledge, thereby obtaining a knowledge-enhanced cross-modal association graph; the cross-modal knowledge enhancement module includes: Obtain a list of knowledge anchor points, which includes multiple sets of knowledge anchor points. Each set of knowledge anchor points includes anchor point ID information, anchor point feature information, and initial verification status. A preliminary collaborative verification unit list is generated based on the knowledge anchor list and the cross-modal association graph. The preliminary collaborative verification unit list includes multiple collaborative verification units, and each collaborative verification unit includes component ID information, anchor ID information, and verification efficiency. The preliminary list of collaborative verification units is optimized to obtain the optimized list of collaborative verification units. Generate adaptive verification configuration based on the optimized list of collaborative verification units; Based on the optimized list of collaborative verification units and the adaptive verification configuration, a knowledge-enhanced cross-modal association graph is generated. A context-aware feature extraction module is used to obtain text semantic vectors, image feature vectors, speech feature vectors, and cross-modal association-enhanced text features based on the knowledge-enhanced cross-modal association graph and text structured data, image structured data, and audio structured data. A dynamic modal weight fusion module is used to obtain modal weight configurations based on the text semantic vector, image feature vector, speech feature vector, and the knowledge-enhanced cross-modal association graph. The process of obtaining the modal weight configurations based on the text semantic vector, image feature vector, speech feature vector, and the knowledge-enhanced cross-modal association graph includes: The basic weights of text, image, and audio are calculated by combining modal vector features and associated graph node attributes. The weights of each mode are adjusted by combining the basic weights of the single mode with the correlation strength coefficient matrix. A contradiction pre-screening module is used to generate a list of potential contradiction points based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, and modal weight configuration. A cross-modal contradiction detection and classification module is used to generate misclassification results based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list.

2. The multimodal AI collaborative processing system for translation and peer review as described in claim 1, characterized in that, The optimization of the preliminary collaborative verification unit list to obtain the optimized collaborative verification unit list includes: The verification quality entropy of each collaborative verification unit in the collaborative verification unit list is calculated by statistical distribution and information entropy, thereby obtaining the verification quality entropy of each collaborative verification unit. Collaborative verification units with verification quality entropy below the threshold are regarded as abnormal units. Adaptive structural reorganization is performed on each anomalous unit to obtain the reorganized unit after reorganization of each anomalous unit; Replace each abnormal unit in the collaborative verification unit list with its corresponding recombination unit to obtain the optimized collaborative verification unit list.

3. The multimodal AI collaborative processing system for translation and peer review as described in claim 2, characterized in that, The verification quality entropy is calculated using the following formula: ; Where K is the number of discretization intervals for the verification efficiency. The probability that the unit efficiency falls in the k-th interval. Let D be the ideal efficiency distribution and D be the dimension of the input features. Let be the second derivative of efficiency with respect to the d-th dimension feature.

4. The multimodal AI collaborative processing system for translation and peer review as described in claim 3, characterized in that, The step of adaptively reorganizing each anomalous unit to obtain the reorganized unit after reorganization includes: Parameter initialization: Initialize the association weight matrix for each anomalous unit; Set initial values ​​for hyperparameters, including: weights for format consistency, efficiency decay, and anchor point diversity. Define the reorganization objective loss function, and use the gradient descent method to solve for each anomalous unit until convergence, thereby obtaining the reorganized unit after reorganization of each anomalous unit.

5. The multimodal AI collaborative processing system for translation and peer review as described in claim 4, characterized in that, The target loss function for recombination is as follows: ； Where α is a hyperparameter of the format consistency term; The feature vector of the term; The feature vector is in the format of; β This is a hyperparameter for the efficiency decay term; The verification efficiency at time τ (dynamic efficiency tracking, τ∈[0,T], where T is the upper limit of the observation period); This is the integral over the time domain; γ For the anchor point diversity term, it is a hyperparameter; C This represents the total number of anchor points. The associated angle of the c-th anchor point; This represents the maximum allowed associated angle.

6. The multimodal AI collaborative processing system for translation and peer review as described in claim 5, characterized in that, The context-aware feature extraction module includes: A text feature extraction module, which is used to generate text semantic vectors based on the text structured data; A visual feature extraction module, which is used to generate an image feature vector based on the image structured data; A cross-modal knowledge enhancement module is used to generate cross-modal association-enhanced text features based on the text structured data, image structured data, and the knowledge-enhanced cross-modal association graph. An audio feature extraction module is used to generate speech feature vectors based on structured audio data.

7. The multimodal AI collaborative processing system for translation and peer review as described in claim 6, characterized in that, The cross-modal contradiction detection and classification module includes: XGBoost model acquisition module, which is used to acquire trained XGBoost models; The classification module is used to input the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list into the trained XGBoost model to obtain misclassification results.

8. A multimodal AI collaborative processing method for translation and peer review, characterized in that, The multimodal AI collaborative processing method for translation peer review includes: Acquire the data to be translated, which includes text data, image data, and audio data to be translated; The text data, image data, and audio data to be translated are processed into structured data to be translated, which includes structured text data, structured image data, and structured audio data. Establishing a cross-modal association map based on the structured data to be translated; the establishment of the cross-modal association map based on the structured data to be translated includes: All nodes of the three modalities of text, image, and audio are aggregated to form a multimodal node list. Each node in the list contains a unique ID, node type, core information, and associated structured data identifier. Based on a multimodal node list, an information interaction efficiency matrix is ​​constructed by calculating the information interaction efficiency between nodes of different modalities. Density clustering algorithm is used to cluster multimodal nodes: with each node as the core, other modal nodes with interaction efficiency values ​​higher than the corresponding threshold are included in the candidate association set; if the number of nodes in the candidate association set is ≥3, the core node and the nodes in the candidate set together form an initial association cluster; if the number of nodes in the candidate association set is <3, the node is marked as a node to be associated. Each initial associated cluster is assigned a unique cluster ID, and records the IDs of all nodes contained in the cluster, the node types, and the maximum interaction efficiency between each node, forming an initial associated cluster list; Determine the core components of the graph: The graph uses multimodal nodes as nodes, the information interaction efficiency between nodes as edges, and the initial association clusters as subgraph units; The hierarchical structure of the graph is constructed as follows: The graph consists of three layers: core layer, association layer, and outer layer. The core layer is the combination of nodes with the highest interaction efficiency in each initial association cluster, the association layer consists of other nodes in the cluster except for the core layer, and the outer layer consists of nodes to be associated. The cross-modal association graph is augmented with knowledge to obtain an augmented cross-modal association graph; the cross-modal knowledge augmentation module includes: Obtain a list of knowledge anchor points, which includes multiple sets of knowledge anchor points. Each set of knowledge anchor points includes anchor point ID information, anchor point feature information, and initial verification status. A preliminary collaborative verification unit list is generated based on the knowledge anchor list and the cross-modal association graph. The preliminary collaborative verification unit list includes multiple collaborative verification units, and each collaborative verification unit includes component ID information, anchor ID information, and verification efficiency. The preliminary list of collaborative verification units is optimized to obtain the optimized list of collaborative verification units. Generate adaptive verification configuration based on the optimized list of collaborative verification units; Based on the optimized list of collaborative verification units and the adaptive verification configuration, a knowledge-enhanced cross-modal association graph is generated. Based on the knowledge-enhanced cross-modal association graph and the structured text data, image data, and audio data, obtain text semantic vectors, image feature vectors, speech feature vectors, and cross-modal association-enhanced text features; The modal weight configuration is obtained based on the text semantic vector, image feature vector, speech feature vector, and the knowledge-enhanced cross-modal association graph; the process of obtaining the modal weight configuration based on the text semantic vector, image feature vector, speech feature vector, and the knowledge-enhanced cross-modal association graph includes: The basic weights of text, image, and audio are calculated by combining modal vector features and associated graph node attributes. The weights of each mode are adjusted by combining the basic weights of the single mode with the correlation strength coefficient matrix. A list of potential contradictions is generated based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, and modal weight configuration. The misclassification result is generated based on the text semantic vector, image feature vector, speech feature vector, knowledge-enhanced cross-modal association graph, modal weight configuration, and potential contradiction point list.

Images