Unified framework-based dynamic adaptation and collaborative extraction method for large models

Abstract

The application provides a large model dynamic adaptation and collaborative extraction method based on a unified framework, which comprises the following steps: preprocessing the multi-modal data to be processed; dynamically identifying the modal type and adapting the coding to obtain the basic features; generating the fusion features through the parallel Transformer encoder combined with the semantic anchor points, synchronously monitoring the feature distortion and correcting the errors; realizing the cross-modal information interaction based on the cross-attention mechanism of the anchor point weight enhancement matrix to generate the cross-modal feature vector; determining the differential fusion weight according to the relationship type between the modes to obtain the global fusion features; detecting the semantic consistency and dynamically adjusting the weight; converting the semantic information and supplementing the fine-grained data to form the preliminary results; verifying and completing the attributes based on the knowledge graph; and generating the multi-granularity extraction results containing the global results and the fine-grained grounding information. The application can realize the efficient adaptation, accurate fusion and high-quality information extraction of multi-modal data.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, and in particular to a method for dynamic adaptation and collaborative extraction of large models based on a unified framework. Background Technology

[0002] With the rapid development of information technology, technological innovations represented by artificial intelligence, 5G communication, and cloud computing are driving the unprecedented growth of multimodal data. Multimodal data encompasses various information carriers such as text, images, audio, and video, with typical forms including images with text annotations and videos with voice narration. This type of data is experiencing explosive growth in fields such as the internet, intelligent monitoring, autonomous driving, medical diagnosis, and educational technology. In the internet sector, social media platforms generate massive amounts of multimodal content daily, combining text, images, videos, and comments; intelligent monitoring systems need to simultaneously process camera footage and ambient sound; and in autonomous driving scenarios, vehicles must analyze sensor images, radar data, and navigation voice commands in real time. These application scenarios place higher demands on multimodal information extraction technologies.

[0003] However, the current development of multimodal information extraction technology lags significantly behind application needs, exhibiting significant technical bottlenecks, specifically in the following key dimensions: Modal adaptability limitations: Traditional multimodal information extraction methods are mostly designed for fixed modal combinations, such as the common "text + image" pattern. When faced with dynamic modal inputs, such as complex combinations of "text + audio + video," the system needs to refactor the encoding logic. This not only increases development costs and time but also results in extremely low adaptability, making it difficult to meet the needs of application scenarios with high real-time requirements.

[0004] Insufficient semantic association mining: Existing technologies for multimodal feature fusion often remain at the surface feature level, making it difficult to delve into the deep semantic associations between different modalities. For example, in traffic accident analysis, there is a close implicit correlation between the "collision intensity" described in the text and the deformation area of ​​the vehicle in the image, as well as the collision decibel value recorded in the audio. However, traditional methods often fail to effectively capture these associations, resulting in incomplete information extraction.

[0005] Lack of error robustness: During information extraction, existing technologies are prone to problems such as content duplication, incomplete information, and grounding mismatch. More seriously, the systems generally lack real-time monitoring and correction mechanisms. Once an error occurs, it will continue to propagate in subsequent processing steps, ultimately causing the extracted results to deviate significantly from the actual situation.

[0006] Rigid Modal Relationship Processing: Different modalities exhibit various relationships, such as "shared" and "independent." When text and images describe the same entity, they are considered to have a "shared" relationship; however, when text and audio content are unrelated, they are considered to have an "independent" relationship. Existing technologies often employ a uniform fusion strategy to process all modal relationships, which is highly susceptible to introducing noise or losing crucial information, thus affecting the accuracy of information extraction.

[0007] Therefore, to meet increasingly complex application demands, there is an urgent need for a multimodal unified information extraction technology capable of adaptively processing dynamic modal inputs, enhancing deep semantic associations, correcting errors in real time, and dynamically adapting modal relationships. The development of this technology will not only overcome existing technological bottlenecks but also provide strong support for the efficient utilization of multimodal data, promoting technological development and application innovation in related fields. Summary of the Invention

[0008] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: This invention provides a method for dynamic adaptation and collaborative extraction of large models based on a unified framework, the method comprising the following steps: S100 performs dynamic modality recognition and adaptation on the multimodal data to be processed, and encodes the features of each modality to obtain the basic features of each modality.

[0009] S200 identifies the semantic anchor points of each modal data, concatenates the basic features and semantic anchor points of each modal data, and inputs them into the corresponding parallel Transformer encoder for encoding to obtain the fused features of each modal data; at the same time, a monitoring node is set at the output of each layer of each encoder to monitor and correct feature distortion in real time.

[0010] S300, based on the fusion features of data from each modality, realizes cross-modal information interaction and outputs cross-modal feature vectors of each modality; S400: Obtain the relationship type between each modal data, then determine the corresponding modal fusion weight based on the obtained relationship type, and perform differential fusion on the cross-modal feature vectors of each modal data to obtain the current global fusion feature.

[0011] S500 checks the semantic consistency of the global fusion features; if a semantic conflict exists, the current fusion weights of each modality are dynamically adjusted and the process returns to S400; otherwise, S600 is executed.

[0012] S600 converts globally fused features without semantic conflicts into semantic information, and supplements fine-grained data according to the fine-grained requirements of the extraction instructions to form preliminary semantic extraction results.

[0013] S700 verifies the completeness of attributes in the preliminary semantic extraction results based on the knowledge graph. If an attribute is missing, a completion process is triggered to obtain a verification result that meets the completeness standard.

[0014] S800, based on the verification result, generates a multi-granularity unified information extraction result that includes global extraction results and fine-grained grounding information.

[0015] The present invention has at least the following beneficial effects: (1) Break through the bottleneck of modal adaptation and significantly improve multimodal compatibility. Through dynamic modal recognition and adaptation design, based on modal feature entropy values ​​and the ImageBind model, this method can automatically identify known modalities (text / image / audio / video) and unknown modalities (such as time-series signals from novel sensors and specially coded text). It dynamically adjusts feature dimensions and encoding strategies according to the physical properties of different modalities (e.g., using Fourier transform to extract frequency domain features for continuous signals and character-level embedding to capture local features for discrete structures). Compared to existing fixed modal adaptation schemes, this method is compatible with at least three types of unknown modalities, improving modal adaptation coverage by over 60%. It solves the technical pain points of traditional methods in "unable to recognize and encode" unknown modalities, meeting the processing needs of multi-source heterogeneous multimodal data in complex scenarios.

[0016] (2) Block the propagation of feature distortion and ensure the stability of feature processing quality. A monitoring node is set at the output of each layer of the parallel Transformer encoder. Normal feature vectors are predicted by the sliding window method (temporal modality) or the mean of similar data (non-temporal modality). Feature distortion is detected by combining the difference formula, triggering a recoding mechanism (temporary storage is performed if three consecutive anomalies occur). This design can intercept feature distortion caused by device noise and transmission packet loss in real time, with a feature distortion correction rate of ≥95%. It prevents distorted features from entering the subsequent fusion process, improves the accuracy of cross-modal feature interaction by more than 40%, and solves the problem of "hidden propagation of feature errors and difficulty in later correction" in traditional methods, ensuring the stability of feature quality throughout the entire process.

[0017] (3) Achieve differentiated cross-modal fusion and improve the accuracy of semantic fusion. Based on modality sharing scores, shared / independent relationships are distinguished. Shared relationships are deeply bound to highly correlated features through a cross-attention mechanism and their weights are optimized using a modality complementarity loss function. Independent relationships are prevented from interference by using weak fusion and a modality isolation mask. Compared to the traditional "unified weight fusion" scheme, this method improves the semantic association capture accuracy of shared relationship modalities by 50% and reduces information interference of independent relationship modalities by 70%. It effectively solves the rigid fusion problem of "insufficient fusion of shared modal information and cross-contamination of independent modalities," achieving a semantic consistency rate of ≥90% for the global fused features. (4) Re-verification ensures result quality and improves the completeness and accuracy of information extraction. A lightweight LLM is used to detect semantic conflicts (temporal / spatial / attribute contradictions) in globally fused features, and weights are dynamically adjusted for iterative optimization (conflict correction rate ≥85%). Attribute integrity is verified based on a knowledge graph (threshold triggers sub-modules to complete missing attributes, such as OCR extracting text attributes and SEEM segmenting visual attributes). This dual verification increases the attribute integrity rate of the initial semantic extraction results from 60% in traditional methods to over 92%, and reduces the semantic conflict rate to below 5%, solving the problems of "semantic contradictions and missing key attributes" in traditional methods, ensuring the usability and reliability of the output information.

[0018] (5) Output multi-granularity results to adapt to diverse application scenarios. This method generates multi-granular results consisting of "global extraction results (entity list / relation triples) + fine-grained grounding information (segmentation mask and entity binding relationship)". The global results meet the needs of "rapid decision-making" scenarios such as intelligent monitoring and early warning, and e-commerce fast retrieval (response speed improved by 30%), while the fine-grained information meets the needs of "precise analysis" scenarios such as accident liability determination and medical lesion tracing (detailed information coverage improved by 80%). Compared with traditional single-format output solutions, this method can adapt to the information usage needs of at least 5 different application scenarios, expanding the application boundaries of multimodal information extraction technology.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 The flowchart illustrates the large-model dynamic adaptation and collaborative extraction method based on a unified framework provided in this embodiment of the invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0024] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0025] This invention aims to address the technical problems existing in current multimodal information extraction technologies, such as poor modality adaptability, weak semantic association, low error robustness, and rigid modality relationship processing. By constructing an innovative method and system for dynamic adaptation and collaborative extraction of large models based on a unified framework, it breaks down barriers between modalities, enhances semantic associations between different modal data, improves the robustness of models in complex data environments, flexibly handles modality relationships, and achieves efficient and accurate information extraction from multimodal data, meeting the urgent needs of various industries for in-depth mining and utilization of multimodal information.

[0026] This invention provides a method for dynamic adaptation and collaborative extraction of large models based on a unified framework, such as... Figure 1 As shown, the method may include the following steps: S100 performs dynamic modality recognition and adaptation on the multimodal data to be processed, and encodes the features of each modality to obtain the basic features of each modality.

[0027] Before S100, the following steps are also included: Step 1: Obtain the raw multimodal data and extraction instructions.

[0028] In this embodiment of the invention, multimodal data refers to a data set containing two or more heterogeneous information carriers. These information carriers have different physical properties and forms of expression, including but not limited to: text (such as natural language descriptions, structured forms), images (such as still pictures, remote sensing images), audio (such as voice clips, environmental sound effects), video (such as continuous frame sequences, surveillance videos), radar point clouds (such as three-dimensional spatial coordinate data), time-series signals (such as electrical signals collected by sensors, temperature change curves), etc.

[0029] Extraction instructions are structured / semi-structured instructions initiated by users, downstream business systems (such as intelligent analysis platforms and automated decision-making systems), or application scenarios to specify the objectives, scope, accuracy, and output requirements for multimodal information extraction. They serve as the core basis for subsequent multimodal data processing (such as dynamic modality recognition and feature encoding) and semantic extraction (such as fine-grained information supplementation and global result generation), aiming to ensure that the extraction process accurately matches the actual application needs. Their key components typically include, but are not limited to: Extraction Target: Identify the core information types to be obtained from multimodal data, such as entities (e.g., "product name" in text, "vehicle target" in images), relationships (e.g., "employee-department" association, "segmentation region-entity description" binding relationship between images and text), events (e.g., event graph of "accident time-location-involved parties"), or specific attributes (e.g., "speaker identity" in audio, "target size" in radar point clouds); Fine-grained requirements: Define the granularity standard for information extraction, such as whether text needs to be extracted to the "phrase level" or "sentence level", whether image segmentation needs to be accurate to the "pixel-level mask" or "target bounding box", and whether time-series signals need to be extracted to the "real-time instantaneous value" or "time period statistical value". Modal association requirements: Specify whether cross-modal association extraction is required (e.g., "bind pedestrians in video frames to speakers in synchronized audio"), or extraction is only for a single modality (e.g., "extract building coordinates only from remote sensing imagery"). Output specifications: Define the format of the extracted results (such as structured tables, knowledge graphs, JSON strings), storage format (such as tensors, text files), and delivery method (such as real-time push, batch export). Accuracy threshold: Set the quality standard for the extraction results, such as entity recognition accuracy ≥95%, cross-modal information matching IoU (Intersection over Union) ≥0.8, etc., which are used as the basis for judgment in subsequent verification stages (such as S700 attribute integrity verification).

[0030] Step 2: Preprocess the original multimodal data to obtain preprocessed multimodal data, which will be used as the multimodal data to be processed.

[0031] In this embodiment of the invention, the preprocessing includes: cleaning data noise using an outlier detection algorithm (such as the Z-score method), removing low-quality samples by sample quality scoring (fusion clarity, completeness, and annotation accuracy), correcting erroneous labels based on manual verification and a rule engine, and finally converting the data into a computable tensor form according to modality type, such as converting text into character index tensors and images into pixel value tensors.

[0032] Furthermore, S100 specifically includes: S110 automatically identifies the modality type of the multimodal data to be processed based on modal feature entropy values ​​and a preset multimodal embedding model.

[0033] The pre-defined multimodal embedding model is the ImageBind model, a multimodal foundational model proposed by Meta. Its core function is to map features from various known modalities, such as text, images, audio, video, depth maps, and heatmaps, to a unified embedding space (i.e., a vector space with the same feature vector dimension) through a pre-defined modality alignment mechanism, achieving semantic comparability of cross-modal features. This model, through a shared feature encoding framework and cross-modal comparative learning, ensures that semantically related content in different modalities has similar vector representations in the embedding space, providing a foundation for unified processing of multimodal data.

[0034] The modal types include known modalities and unknown modalities. Known modalities refer to modal types natively supported by the ImageBind model, including text, images, audio, video, radar point clouds, depth maps, etc. Unknown modalities refer to novel information carriers not natively supported by the ImageBind model, including but not limited to time-series electrical signals from novel sensors (such as millimeter-wave radar-derived signals), text strings with novel encoding formats (such as encrypted text with custom protocols), and biometric signals (such as heart rate waveforms).

[0035] In this embodiment of the invention, the modal feature entropy value H(M) is used to quantify the complexity of the modal feature distribution, and the calculation formula is: H(M) = ∑ q j=1 (p) j ×log2p j ).

[0036] Where M represents the modal data to be identified, p j Let H(M) be the probability of the j-th type of feature in this modality (e.g., the proportion of features such as "high-frequency energy" and "low-frequency energy" in an audio modality), and let q be the total number of feature categories in this modality. When H(M) is greater than H0, it is determined to be an unknown modality (the feature distribution is complex and significantly different from known modalities); when H(M) ≤ 1.2, the specific known modality type is determined by matching with the known modality feature library of the ImageBind model. H0 is a preset entropy threshold, which can be 1.2.

[0037] S120, perform feature encoding and unified embedding processing on the identified known and unknown modalities respectively to obtain the initial features of the known and unknown modalities respectively.

[0038] For known modalities: First, based on the physical properties of each modality (such as the semantic sequence of text, the spatial structure of images, and the temporal fluctuation of audio), a dedicated encoder adapted to that modality is used to extract high-dimensional features; then, through the modality adapter of the ImageBind model, the dedicated features of different dimensions are uniformly mapped to a shared embedding space of a preset dimension, such as 1024 dimensions, to form semantically comparable initial features, laying the foundation for subsequent cross-modal interactions.

[0039] Specifically, for text modalities: pre-trained language models (such as BERT-base, RoBERTa) are used to encode text sequences (such as natural language sentences, structured form fields), and the semantic relationships between words are captured through a contextual attention mechanism, outputting a 768-dimensional word embedding vector sequence (each vector corresponds to a word or sub-word in the text); the 768-dimensional word embedding vectors are input into the text adapter of the ImageBind model, and through linear transformation and feature alignment, they are converted into 1024-dimensional unified embedding features, preserving the semantic logical relationships of the text.

[0040] For image modalities: Convolutional neural networks (such as ResNet-50 and ViT) are used to extract features from static images (such as photographs and remote sensing images). Multi-layer convolution / self-attention operations are used to capture the local texture, global structure, and semantic information (such as object contours and scene types) of the image, and output a 2048-dimensional global visual feature vector. The 2048-dimensional visual features are input into the image adapter of the ImageBind model and mapped to a 1024-dimensional unified embedding space through non-linear transformation to ensure that the image features of the same object / scene are adjacent to the corresponding text description features in the embedding space.

[0041] For audio modalities: First, the audio signal (such as speech, ambient sound effects) is preprocessed (sampling rate is uniformly set to 16kHz, duration is truncated / pasted to 3 seconds), and a Mel spectrogram (128×128 dimension) is generated through short-time Fourier transform; then, an audio convolutional network (such as PANNs, AudioCNN) is used to encode the Mel spectrogram and extract 128-dimensional temporal-spectral joint features (including pitch, loudness, rhythm, etc.); then, the 128-dimensional audio features are input into the audio adapter of the ImageBind model, and after dimensional expansion and semantic alignment, they are converted into 1024-dimensional unified embedding features to achieve cross-modal association of "sound-semantics" (such as the proximity of the audio feature of "car horn" to the text feature of "car").

[0042] For video modalities: a keyframe selection mechanism (such as K-means clustering based on inter-frame difference) is used to extract 16 keyframes (covering the core content of the video) from the video sequence (such as surveillance video, short video); for each keyframe, the ResNet-50 encoder of the image modality is reused to extract 2048-dimensional visual features, and then the temporal correlation between frames is captured by a temporal attention network (such as BiLSTM) to output 2048-dimensional global video features; the 2048-dimensional video features are input into the video adapter of the ImageBind model, and combined with temporal information, they are weighted and transformed into 1024-dimensional unified embedding features to ensure that the dynamic content of the video (such as "vehicle driving") and the corresponding text description are semantically consistent in the embedding space.

[0043] For radar point cloud modalities: The radar point cloud data (such as the 3D spatial point set acquired by lidar) is preprocessed (denoising and downsampling to 1024 points). A point cloud convolutional network (such as PointNet++, PointTransformer) is used to encode the 3D coordinates (x, y, z) and reflection intensity features. Through local neighborhood aggregation and global feature pooling, 1024-dimensional spatial structure features (including information such as object shape and positional relationships) are output. The 1024-dimensional point cloud features are input into the point cloud adapter of the ImageBind model. After spatial and semantic feature alignment, the 1024-dimensional unified embedding dimension is maintained to achieve cross-modal association of "3D structure-semantic description" (such as the proximity of "obstacle point cloud" and "obstacle" text features).

[0044] For the depth map modality: The depth map (a single-channel image that records pixel distance information) is preprocessed (depth values ​​are normalized and the size is unified to 224×224). A depth-aware convolutional network (such as DenseDepth, DepthNet) is used to extract features such as depth gradient and object contour. By fusing local depth differences with global scene structure, a 512-dimensional depth feature vector is output. The 512-dimensional depth features are input into the depth map adapter of the ImageBind model and transformed into a 1024-dimensional unified embedding feature through dimensional expansion (supplementing semantic information through residual connections). This enables the depth information (such as "object height" and "distance") to semantically match with the corresponding text / image features in the embedding space.

[0045] For unknown modalities: the corresponding feature extraction submodule is triggered based on the physical properties of the unknown modalities. After performing preliminary feature extraction, it is mapped to the unified embedding space of the ImageBind model to form the initial features of the unknown modalities.

[0046] Among these methods, the unknown mode can be determined as a continuous signal or a discrete structure by parsing the data format, such as whether it contains a timestamp or is a continuous numerical sequence.

[0047] Specifically, based on the physical properties of the unknown modality, a corresponding feature extraction submodule is triggered. After performing preliminary feature extraction, the features are mapped to the unified embedding space of a preset multimodal embedding model to form the initial features of the unknown modality, which specifically include: If the unknown mode is a continuous signal, such as the time-series electrical signal of a novel sensor, the submodule will automatically enable Fourier transform to extract frequency domain features and wavelet analysis to extract joint time-frequency domain features, outputting a 512-dimensional feature vector. If the unknown modality is a discrete structure such as a text string with a new encoding format, then local structural features are captured by methods such as character-level embedding (CharCNN) and n-gram (n=2,3) statistics, and a 512-dimensional feature vector is output. By training an adaptation layer (with known modality embeddings as supervision), 512-dimensional features are mapped to the 1024-dimensional unified embedding space of the ImageBind model, forming the initial features of the unknown modality.

[0048] This design does not require pre-setting modal types; it only triggers the corresponding processing logic based on the physical properties of the data (such as whether it is a time-series signal or whether it contains spatial coordinates), thereby achieving undifferentiated preliminary feature extraction of unknown modalities.

[0049] S130, dynamically adjust the initial dimensions of the known and unknown modalities to obtain a standard tensor in a preset format, which is the basic feature of each modal data.

[0050] Specifically, for modalities lacking a temporal dimension (such as static images and text), the temporal dimension is supplemented by feature copying (copying feature vectors to a preset temporal length); for modalities lacking a spatial dimension (such as audio and temporal signals), virtual spatial coordinates are generated through interpolation (constructing spatial topology based on feature similarity); for modalities with redundant dimensions, such as long videos (duration > 30s) and high-resolution images (resolution > 4K), a self-attention mechanism (such as TransformerEncoder) is used to focus on key feature regions, retaining core information while compressing to a preset dimension; the final output is a standard tensor with a uniform format, whose dimensions are [batch_size, feature_dim=1024, Sequence_Len=32, Spatial_H=16, Spatial_W=16], ensuring that it can be directly input into subsequent parallel Transformer encoders for processing.

[0051] S200 identifies the semantic anchor points of each modal data, concatenates the basic features and semantic anchor points of each modal data, and inputs them into the corresponding parallel Transformer encoder for encoding to obtain the fused features of each modal data; at the same time, a monitoring node is set at the output of each layer of each encoder to monitor and correct feature distortion in real time.

[0052] In this embodiment of the invention, semantic anchors refer to key units in each modality of data that carry core semantic information (such as entities in text, core objects in images, and landmark events in audio). Their function is to provide a "semantic benchmark" for cross-modal information interaction and ensure feature alignment of different modalities on the same semantic dimension.

[0053] The identification and feature extraction of semantic anchors for each modality follow the logic of "modality-specific recognizer + unified dimension embedding", as follows: Text Modality: Entity recognition models (such as spaCy, BERT-NER) are used to identify core entities (such as names of people, names of events, time, location, etc.) and semantic keywords (such as verbs, adjectives) in the text as semantic anchors; the anchors are converted into 768-dimensional embedding vectors through pre-trained language models (such as BERT) to preserve the semantic attributes of the anchors (such as entity type, semantic role).

[0054] Image modality: Key regions (such as main objects and scene landmarks) in the image are identified using object detection models (such as YOLOv8 and Faster R-CNN) as semantic anchors; the anchor regions (image sub-regions cropped by bounding boxes) are encoded by a visual Transformer (such as ViT) to output a 768-dimensional visual anchor embedding vector, which includes features such as the shape, color, and position of the anchors.

[0055] Audio modality: Audio event detection models (such as PANNs and YAMNet) are used to identify characteristic events in the audio (such as voice commands, specific sound effects, and environmental landmark sounds) as semantic anchors; Mel spectrum analysis is performed on the audio segments corresponding to the anchors (such as 1-3 second speech segments), and 128-dimensional temporal features are extracted through audio Transformer and then mapped to 768-dimensional audio anchor embedding vectors.

[0056] Video Modal: Combining Temporal and Spatial Anchors—Key action segments in the video (such as "vehicle turning" or "person raising hand") are identified using a temporal action detection model (such as SlowFast) as temporal anchors, and core objects in the segments are identified using an object detection model as spatial anchors; after fusing the temporal and spatial features, they are converted into a 768-dimensional video anchor embedding vector.

[0057] Radar point cloud modality: A point cloud segmentation model (such as PointRCNN) is used to identify key targets (such as pedestrians, vehicles, and obstacles) in the point cloud as semantic anchors; the feature aggregation of the point set corresponding to the anchor (the sub-point cloud extracted by clustering) is performed to output a 768-dimensional point cloud anchor embedding vector, which contains information such as the target's three-dimensional structure and motion state.

[0058] Depth map modality: A depth feature segmentation model (such as Depth-AwareCNN) is used to identify key depth regions (such as "near objects" and "distant boundaries") in the depth map as semantic anchors; features such as depth gradient and distance distribution of the anchor regions are extracted and converted into 768-dimensional depth anchor embedding vectors.

[0059] In this embodiment of the invention, the parallel Transformer encoder adopts a "multi-branch independent encoding + cross-modal interactive fusion" architecture, which includes an input layer, an encoding layer, and an interaction layer. Each modality corresponds to an independent encoder branch, and the branches are associated with each other through the interaction layer, as detailed below: (1) Input layer: Feature concatenation and dimension alignment The "basic features" (standard tensors output by S100, with dimensions [batch_size, 1024, 32, 16, 16]) and "semantic anchor features" (768-dimensional embedding vectors) of each modality are processed as follows: Dimension alignment: The 768-dimensional anchor embedding vector is expanded to 1024 dimensions through linear mapping to match the feature dimensions of the base features; Feature concatenation: The aligned anchor features are concatenated with the basic features according to the channel dimension to generate the final input tensor of each modality, with dimensions of [batch_size,2048,32,16,16] (1024-dimensional basic features + 1024-dimensional anchor features), ensuring that the encoder receives both global features and core semantic anchor information simultaneously.

[0060] (2) Coding layer: Modality-independent deep coding Each modality encoder branch operates independently, taking a tensor (dimension [batch_size, 2048, 32, 16, 16]) concatenated with "basic features + semantic anchor features" as input. Deep encoding is used to organically fuse these two parts of information, ultimately outputting the fused features of each modality. A single branch contains N=6 encoder blocks (expandable to 12 layers). The structure and function of each encoder block are designed around "feature fusion," as detailed below: Location Encoding: Temporal / spatial location information is injected into the concatenated features, simultaneously covering the positional correlation between basic features and semantic anchor features. For temporal modalities such as text / audio / video, the temporal position of basic features (e.g., text sequence position, audio frame order) and the temporal coordinates of semantic anchors (e.g., the position of entity words in text, the timestamp of events in audio) are simultaneously marked in the temporal dimension. For spatial modalities such as images / point clouds / depth maps, the spatial distribution of basic features (e.g., pixel coordinates, 3D position of point clouds) and the spatial boundaries of semantic anchors (e.g., object detection box coordinates, key region contours) are simultaneously encoded in the spatial dimension, ensuring that the model simultaneously captures both the "positional correlation of basic features themselves" and the "positional correspondence between basic features and anchor features." Self-Attention Mechanism: The association weights within features are calculated using Query(Q), Key(K), and Value(V) (e.g., the association between "event" and "time" in text, and the association between "object" and "background" in an image). The formula is: Self-Attention(Q, K, V) = softmax(QK). T / (d k ) 1 / 2 )V.

[0061] Where d k The Q / K vector dimension (with a value of 512) focuses on key features within the modality through self-attention. Q, K, and V are all derived from a concatenated tensor of "basic features + semantic anchor features". Attention calculation not only focuses on key features within the modality (such as the association between "event" and "time" in text, and the association between "object" and "background" in image), but also focuses on capturing the binding relationship between "basic features and semantic anchors" (such as the association between the "accident" description in text basic features and the anchor "collision" entity, and the association between "blurred region" in image basic features and the anchor "vehicle" target). The semantic constraints of anchors on basic features are strengthened through weight allocation.

[0062] Feedforward Neural Network: Composed of two layers of linear transformation (with the middle dimension set to 2048) and ReLU activation function, it performs non-linear mapping on the "basic feature-anchor feature" association features of the self-attention output, and further integrates the deep semantics of the two parts of information (such as transforming the association information of the "vehicle outline" basic features and the "accident vehicle" anchor features in the image into the more abstract "accident subject" semantic features), thereby enhancing the expressive power of the fused features.

[0063] Residual connections and layer normalization: After each operation, residual connections are added (adding the input features to the output features of the current layer) to avoid information loss during the fusion of "basic features and anchor features". At the same time, layer normalization stabilizes the feature distribution, ensuring that after 6 layers of encoding, the basic features and semantic anchor features in the output feature tensor have been deeply fused, forming a modal fusion feature that combines global information and core semantic benchmarks (dimensionality maintained [batch_size,2048,32,16,16]), laying the foundation for cross-modal association in subsequent interaction layers.

[0064] (3) Interaction layer: Enhanced cross-modal anchor point association The interaction layer includes a cross-attention information fusion module, which strengthens the correlation between semantic anchors of different modalities through an "anchor weight enhancement matrix" to achieve cross-modal deep semantic alignment. The specific mechanism is as follows: Cross-attention computation: For any two modalities (such as text and image), using the features of modality A as Query (Q) and the semantic anchor features of modality B as Key (K) and Value (V), the cross-modal attention computation mechanism outputs the directional attention result of modality A to modality B. The cross-modal attention computation mechanism is as follows: Output AB =softmax((QK) T ⊙W a ) / (d k ) 1 / 2 )・V.

[0065] Where: Output AB W represents the unidirectional cross-attention result of mode A on mode B. a This is the anchor weight enhancement matrix (with the same dimension as K), where the diagonal elements take values ​​of 1.2-1.3 (enhancing the weight of the semantic anchor itself), and the off-diagonal elements take values ​​of 1 (maintaining the basic weight of non-anchor features). This matrix amplifies the role of anchors in cross-modal association (such as the association between the text "car" entity and the image "car" target); ⊙ indicates element-wise multiplication, and ・ indicates matrix multiplication. V is weighted with normalized attention weights, and the output is a cross-modal feature vector of modality A with the same dimension as Q.

[0066] Multimodal parallel interaction: All modal branches interact in pairs through the above cross-attention mechanism (such as text ↔ image, audio ↔ video, etc.), and finally output cross-modal feature vectors that integrate cross-modal association information of each modal data.

[0067] Furthermore, in the S200, a monitoring node is set at the output of each layer of each encoder. Through a closed-loop mechanism of "normal feature baseline construction → real-time difference detection → hierarchical error correction and anomaly handling", precise interception and correction of feature distortion is achieved, specifically including: S210, for each encoder, determine the normal feature vector of the current frame feature vector according to its corresponding coding mode type; wherein, if the coding mode of the encoder is a temporal mode, the normal feature vector of the current frame is calculated by the sliding window method; if it is a non-temporal mode, the feature mean of the same type of actual data in the application scenario is used as the normal feature vector of the current frame.

[0068] In this embodiment of the invention, temporal modality refers to a mode with temporal continuity (such as audio, video frame sequences, radar point cloud temporal data, and sensor dynamic signals), and its normal feature vector is generated based on "historical feature smoothing values ​​within a sliding window". The sliding window size is set to 5 frames (which can be dynamically adjusted according to the modality sampling rate, such as 10 frames for high-frequency sensor data). The window contains the historical features of the previous 4 frames of the current frame (avoiding the inclusion of the current frame itself to prevent contamination of the baseline).

[0069] In this embodiment of the invention, the normal feature vector F t-normal =(1 / k)∑ t-1 r=t-k F r F r Let be the historical feature vector of the r-th frame, where r ranges from tk to t-1, k is the window size, and t is the current frame number. When t ≤ k (e.g., the first 5 frames), the mean feature value of the previous t-1 frames is fused with the mean feature value of the initial features of similar samples to generate a baseline, thus avoiding the problem of a blank window with no historical data at t=1.

[0070] Non-temporal modalities refer to static modalities without temporal correlation (such as text, single images, and single-frame depth maps), whose normal feature vectors are generated based on the "feature distribution of samples of the same type in the same scene": Pre-built scenario feature library: Classified by application scenario (such as traffic monitoring, medical diagnosis), high-quality samples of non-temporal modalities in each scenario (manually verified to be distortion-free) are collected, and the feature mean and variance of samples of the same type are calculated to form a scenario benchmark library; Normal feature vectors directly call the mean feature value of similar samples in the current scene: F normal =Avg scene，type Avg scene，type This represents the feature mean of this modality type (e.g., "accident scene image") in the current scenario (e.g., "traffic accident").

[0071] S220: Calculate the difference between the current frame feature vector and the normal feature vector at the current encoder output. If the difference is greater than a preset threshold, determine that the current frame feature vector has feature distortion.

[0072] In this embodiment of the invention, the difference D between the current frame feature vector and the normal feature vector satisfies the following condition: D = (1 / d)∑d s=1 |F normal，s -F current，s |, where F normal，s F is the s-th element in the normal feature vector. current，s This is the s-th element of the feature vector output by the current encoder, where s takes values ​​from 1 to d, and d is the dimension of the feature vector, for example, 1024 dimensions.

[0073] In one embodiment of the present invention, the preset threshold can be a fixed value, such as 0.15.

[0074] In another embodiment of the present invention, the preset threshold can be dynamically adjusted according to the modality type and feature level (different modalities / different coding layers have different feature stability): Temporal modalities (such as audio and video): The threshold for the bottom coding layer (first 3 layers) is set to 0.2 (features contain more details, allowing for greater fluctuations), and the threshold for the top coding layer (last 3 layers) is set to 0.15 (features are more abstract, requiring strict control of fluctuations). Non-temporal modalities (such as text and images): The threshold for the bottom coding layer is set to 0.18, and the threshold for the top coding layer is set to 0.12 (static features have higher stability, so the threshold is more stringent).

[0075] S230, if feature distortion is detected, an adaptive error correction mechanism is immediately triggered. If the difference value of the feature vector of the current frame is still greater than the preset threshold after triggering the error correction mechanism for m consecutive times, for example 3 times, the multimodal data to be processed corresponding to the encoder is marked as abnormal data and temporarily stored in the abnormal database. The subsequent processing of the abnormal data is suspended, and the processing is restarted after the data quality is manually reviewed and confirmed.

[0076] In this embodiment of the invention, different error correction mechanisms are adopted according to the degree of distortion to avoid efficiency loss caused by single recoding. Specifically, S230 includes: S2301, if D0<D≤h×D0, execute S2302, if D>h×D0, execute S2303; where D0 is a preset threshold and h is a preset value, for example h=1.5.

[0077] S2302, an error correction mechanism that triggers intra-layer residual correction instead of direct recoding, includes: calculating the residual Δ between the current layer features and the previous layer's output features; and correcting the current features using the residual compensation: F corrected =F current -0.5△ is used to weaken abnormal fluctuations; D is recalculated after correction. If D≤D0, the subsequent processing continues; if D>D0, S2302 is executed.

[0078] S2303, triggers a re-encoding mechanism, including: First re-encoding: The backup encoder is used to re-encode the original data, in which the text modality is replaced with RoBERTa, the image modality is replaced with Swin-Transformer, the audio modality is replaced with YAMNet, etc., to generate new features F1; the difference value D1 of F1 is calculated. If D1≤D0, the candidate processing continues; otherwise, the next re-encoding is performed.

[0079] Second recoding: Adjust the encoding parameters to optimize feature extraction. For example, the compression rate of the image is reduced to 80% of the original resolution, the sampling rate of the audio is increased to 44.1kHz, and the text retains more context windows (e.g., expanded from 512 tokens to 1024 tokens). New features F2 are generated based on the new parameters. The difference value D2 of F2 is calculated. If D2≤D0, the candidate processing continues; otherwise, the next recoding is performed.

[0080] The g-th recoding (2 < g ≤ m): A modal complementarity substitution strategy is adopted. If the current modality is video frame distortion, radar point cloud / audio features with the same timestamp are called to assist in correcting the encoding logic. If it is text distortion, the encoding direction is constrained by the semantic anchor points of the associated image. A new feature Fg is generated and the difference value Dg of Fg is calculated. If Dg ≤ D0, the candidate processing continues. If Dg > D0 and g < m, g = g + 1 is set and this step is repeated. If Dg > D0 and g = m, that is, after m consecutive recodings, the final difference value Dm is greater than the preset threshold, it is determined that it cannot be repaired by automatic error correction, and S2304 is executed.

[0081] S2304, perform the following operations: The data unit is automatically marked as "abnormal" and key information is recorded, including the type of distortion (such as sensor noise, transmission packet loss, data corruption), the D value of each time, and the index of the dimension of the abnormal feature (such as high-frequency audio bands, image edge regions). The data is temporarily stored in the anomaly database, and the subsequent processing of this data unit is suspended (to avoid distortion features from contaminating the cross-modal fusion process). At the same time, multi-level alarms are triggered (real-time system message notifications + daily anomaly summary emails). After manual review, the data is processed according to the scenario: if it is determined to be "repairable" (such as mild noise pollution), a special tool (Webrtcvad for audio noise reduction and LaMa model for image restoration) is called to process it, and then the entire process starting from S100 is restarted; if it is determined to be "unrepairable" (such as data frame loss or severe damage), it is marked as "invalid data" and archived, and the associated log is used for subsequent modal acquisition equipment optimization reference.

[0082] S300 enables cross-modal information interaction based on the fusion features of data from various modalities, and outputs cross-modal feature vectors for each modality.

[0083] This step is used in the interaction layer of the parallel Transformer encoder to realize cross-modal information interaction based on the fusion features of each modality's data, and output cross-modal feature vectors of each modality.

[0084] The input to the interaction layer is the fusion features of various modal data output from the S200 encoding layer. These features share the following commonalities and characteristics: Commonalities: They all contain deep fusion results of basic features + semantic anchor features (e.g., text fusion features contain "original text semantics + entity / event anchors", image fusion features contain "pixel features + target / region anchors"), and the dimension is uniformly d. feat =1024 dimensions (facilitating cross-modal computation); Features: The fusion features of temporal modalities (text, audio, video) include temporal position encoding (such as the sequence position of text tokens and the timestamps of audio frames), and the feature dimensions are [T,d]. feat ], where T is the temporal length; the fusion features of spatial modalities (image, point cloud, depth map) include spatial location encoding (such as pixel coordinates, 3D coordinates of point cloud), and the feature dimension is [H×W, d feat [H×W is the spatial flattening dimension].

[0085] Furthermore, the S300 specifically includes: S301. For any modal data X, its fused features are used as a query vector, and unidirectional cross-attention calculations are performed on the semantic anchor features of other modal data in the multimodal data to be processed, respectively, to generate unidirectional cross-attention results of modal data X for each other modal data.

[0086] The unidirectional cross-attention result of modal data X for each other modal data can be obtained through the aforementioned cross-modal attention mechanism.

[0087] S302, weighted fusion of the modal data X’s own fusion features with all the unidirectional cross-attention results generated in S301, generates cross-modal intermediate features.

[0088] Among them, the cross-modal intermediate feature Mid of modal data X X The following conditions must be met: Mid X =α•F x +∑ u b=1 β h •Output Xh .

[0089] Where, α+∑ u b=1 β h =1. α is the fusion feature F of the modal data X. X The weights, α≥0.4, β hThese are learnable parameters, obtained through model training and optimization. The value of h ranges from 1 to u, where u is the number of other modes. Output Xh This represents the unidirectional cross-attention result of X on the h-th other modal data.

[0090] S303 performs intramodal self-attention optimization on the intermediate features of modality X, resolves cross-modal information conflicts, and outputs the cross-modal feature vector of modality X.

[0091] Intermediate features may exhibit cross-modal information conflicts (e.g., text obtained from an image stating "accident time is 8:00", and from audio stating "8:05"). Conflict resolution and consistency enhancement require intramodal self-attention layers, including: (1) Using the intermediate feature Mid of mode X X Simultaneously serving as Q, K, and V, self-attention optimization features of modal data X are computed through a self-attention mechanism. The self-attention mechanism learns Mid... X The association weights of each feature unit within the model allow the model to autonomously focus on more reliable information, such as prioritizing the adoption of the image's "timestamp anchor" at 8:00, and correcting conflicts with audio.

[0092] (2) Perform layer normalization on the self-attention optimization features of modal data X, and then normalize them with Mid. X After performing residual connection, the cross-modal feature vector of modal data X is obtained.

[0093] The final output cross-modal feature vectors of each mode have the following core characteristics: Preserve its own modal characteristics: Through the weight design of α≥0.4, ensure that the core information of the coding layer fusion features is included (such as the semantic logic of the text and the visual details of the image). Integrating cross-modal associations: Through unidirectional cross-attention computation and weighted fusion, semantic anchor association information with all other modalities (such as the "event-visual" association between text and images) is integrated. Strong semantic consistency: Through self-attention optimization, cross-modal information conflicts are effectively resolved, and the internal logic of the features is self-consistent, providing high-quality input for subsequent cross-modal fusion tasks.

[0094] S400: Obtain the relationship type between each modal data, then determine the corresponding modal fusion weight based on the obtained relationship type, and perform differential fusion on the cross-modal feature vectors of each modal data to obtain the current global fusion feature.

[0095] Furthermore, the S400 specifically includes: S410, obtain the sharing score S(M1, M2) between any two modalities. If S(M1, M2) ≥ S0, it means that the two modalities are closely related in terms of semantics and structure, such as text and image describing the same event. The two modalities are determined to be in a sharing relationship. Otherwise, it means that the relationship is weak, such as background audio and foreground text. They are determined to be in an independent relationship.

[0096] Where S(M1, M2) = α × Sim(V1, V2) + (1-α) × C(M1, M2), where M1 is one of the two modalities, M2 is the other of the two modalities, V1 is the cross-modal feature vector of M1, V2 is the cross-modal feature vector of M2, Sim(V1, V2) is the semantic vector cosine similarity between M1 and M2, and C(M1, M2) is the modal temporal consistency or logical structure consistency score between M1 and M2. 2 represents temporal modalities (such as audio and video), and C(M1,M2) represents the timestamp overlap (ranging from 0 to 1, with higher overlap resulting in higher scores). If M1 and M2 represent spatial or mixed modalities (such as text and images), C(M1,M2) represents the logical structure matching degree (such as the hierarchical correspondence between text entities and image targets, ranging from 0 to 1). S0 represents the preset sharing threshold, for example, S0=0.6, and α represents the weight coefficient, 0<α<1. Through training optimization, the influence of semantic and structural associations is balanced.

[0097] S420, For two modalities determined to have a sharing relationship, the following steps are used to determine the weights of M1 absorbing information from M2 and the weights of M2 absorbing information from M1, which are then used to generate the global association features for each of the two modalities: S4201, based on the semantic attributes (such as event time, location, and subject) of the core related entity (such as "vehicle chase event") that both modalities point to, select directly related feature subsets (such as "chasing" entity features in text and "vehicle movement trajectory" features in image) from V1 and V2, and remove redundant noise features (such as irrelevant auxiliary words in text and background sky features in image) to obtain the selected features VF1 and VF2 (with the same dimension as the original features and noise positions set to 0).

[0098] S4202 uses a cross-attention mechanism to calculate the correlation between feature pairs in VF1 and VF2, specifically including: (1) Using VF1 as Query (Q) and VF2 as Key (K) and Value (V), calculate the first set of association weights Attn1. Attn1[i1][i2] represents the degree of attention of the i1th sub-feature of VF1 to the i2th sub-feature of VF2. Here, i1 is the sub-feature index of VF1 and i2 is the sub-feature index of VF2. The values ​​range from 1 to the feature dimension n. (2) Using VF2 as Query (Q) and VF1 as Key (K) and Value (V), calculate the second set of association weights Attn2, where Attn2[i2][i1] represents the degree of attention paid by the i2th sub-feature of VF2 to the i1th sub-feature of VF1; (3) The correlation degree of the feature pair formed by the i1th sub-feature in VF1 and the i2th sub-feature in VF2 is R(i1, i2) = (Attn1[i1][i2] + Attn2[i2][i1]) / 2.

[0099] S4203 assigns initial weights to the two modalities based on R(i1, i2) to absorb information from each other: The initial weight W1[i] for M1 to absorb information from M2 is positively correlated with the maximum correlation of VF1[i1] in VF2, i.e., W1[i] = max i2 (R(i1, i2)); The initial weight W2[i2] for M2 to absorb information from M1 is positively correlated with the maximum correlation of VF2[i2] in VF1, i.e., W2[i2] = max i1 (R(i1, i2)); Normalize W1 and W2 respectively (weight sum equals 1) to obtain the initial fusion weight vector W1. init and W2 init .

[0100] S4204 aims to minimize the modal complementarity loss function by adjusting the initial weights in reverse to ensure that the complementary features of the two modes (such as features that VF1 has but VF2 does not) are not weakened.

[0101] The modal complementarity loss function satisfies the following condition: L comp =∑ n i=1 |f1(i)-f2(i)|•I(f1(i)≠0∨f2(i)≠0), where f1(i) is the i-th element of the feature vector of one mode in the sharing relation after processing by S4201, f2(i) is the i-th element of the feature vector of another mode in the sharing relation after processing by S4201, n is the dimension of the feature vector, I(•) is the indicator function, and ∨ is the logical OR operator. When f1(i) or f2(i) is non-zero, I(•) takes the value 1, otherwise it is 0.

[0102] The optimization process is as follows: minimize L using gradient descent. comp Adjust W1 init and W2 init The optimized fusion weight W1 is obtained. opt and W2 opt .

[0103] S4205, based on W1 opt and W2 opt Update the features of M1 and M2 respectively (incorporating the other's association information), where the association feature of M1 is updated to: V1 update =V1+W1 opt The association features of V2 and M2 are updated to: V2 update =V2+W2 opt ⊙V1.

[0104] S430, for two modalities M1 and M2 that are determined to be independent, a weak fusion strategy is adopted to avoid interference from irrelevant information and update the correlation features of the two modalities: V1 and V2 remain in an independent encoding state and do not perform cross-modal feature deep interaction (such as not performing cross attention calculation). Both the weight of M1 absorbing information from M2 and the weight of M2 absorbing information from M1 decrease according to the following formula: W weak =W0×(1-S(M1, M2)) is calculated, where W weak W0 is the modal fusion weight, and it can be 0.3. In the weight vector, elements that are cross-related to two independent modalities (such as the weights corresponding to sub-features in M1 that are unrelated to M2) are set to 0 to prevent contamination by irrelevant features. Based on W weak Update the features of M1 and M2 respectively, where the associated features of M1 are updated as follows: V1 update =V1+W weak The association features of V2 and M2 are updated to: V2 update =V2+W weak ⊙V1. S440, for each modality Mx, generate global association features of Mx by interacting with and cumulatively updating all other modalities My (y≠x) in the multimodal data to be processed. Specific steps: S4401, take the cross-modal feature vector Vx of Mx as the initial feature, denoted as Vx current =Vx; S4402, for each other modality My, traverse in a preset order, such as My1, My2, ..., Myu: If Mx and My are shared, call Vx output by S4205. update That is, Vx update =Vx current +Wx opt ⊙Vy, Wx opt Mx is the optimized weight that absorbs My, and Vy is the cross-modal feature vector of My; If Mx and My are independent, call the Vx output by S430.update That is, Vx update =Vx current +Wx weak ⊙Vy, Wx weak The weak fusion weights of My are absorbed by Mx; Update current feature: Vx current =Vx update (Integrate My information into the features of Mx); S4403, Determine global association characteristics: After Mx has completed its interaction with all My, Vx... current That is, the global association feature of Mx contains association information with all modalities, and its dimension is consistent with that of Vx.

[0105] If Mx shares a relationship with other modes My (y≠x), then the output Vx of S4205 is used. update As the update result of the association between Mx and My, if they are independent, the Vx output by S430 is used. update The association update result between Mx and My is used as the association update result between Mx and all other modalities as the global association feature of Mx.

[0106] S450, concatenate the global correlation features of all modalities in a preset order (e.g., temporal modality → spatial modality) to obtain a dimension of [T]. total ,d feat The splicing characteristics of ], T total d is the sum of the feature lengths of all modalities. feat =1024; and then the spliced ​​features are uniformly distributed through a linear transformation layer (such as a fully connected layer or a 1×1 convolution), and finally the current global fusion features are output.

[0107] S500 checks the semantic consistency of the global fusion features; if a semantic conflict exists, the current fusion weights of each modality are dynamically adjusted and the process returns to S400; otherwise, S600 is executed.

[0108] In this embodiment of the invention, globally fused features are input into a lightweight large language model (LLM) (such as distilled Vicuna-7B), and the model is guided by a prompt to detect semantic conflicts between modalities, specifically including: Input definition: Global fusion feature F global The data is then converted into a natural language description sequence via linear projection (e.g., "Text: 'Vehicle collision at 8:00'; Image: 'Collision area at intersection'; Audio: 'Brake sound at 8:05'"). Prompt Design: The fixed template is "Please determine whether there is semantic conflict in the following multimodal information (such as inconsistency in time, location, and event subject), and output the conflict score C(t), which ranges from -1 to 1 (-1 indicates complete conflict, 1 indicates complete consistency, and 0 indicates no obvious conflict): {input multimodal description sequence}". Output: LLM outputs the semantic conflict score C(t) at the current time t (t is the iteration number, initially t=1), used to determine whether a conflict exists. If C(t) < 0: a semantic conflict is determined to exist (the lower the score, the more serious the conflict). If C(t)≥0: the semantics are consistent and no adjustment is needed.

[0109] In S500, if a semantic conflict is detected in the global fusion features, the fusion weights of each modality are adjusted as follows: Wt d =Wt×(1 / (1+e -λ×C（t） Wt represents the modal fusion weights at the current time t, including the shared relation optimization weights output by S4204 and the independent relation weak fusion weights output by S430. d Here, e is the adjusted modal fusion weight for the current time t, λ is the natural constant, and λ is the adjustment coefficient, which can be 5. Experiments verify that when C(t) = -1 (complete conflict), (1 / (1+e)) -λ×C（t） When C(t) ≈ 0.0067, the weight is reduced to 0.67% of the original weight; when C(t) = -0.5 (moderate conflict), (1 / (1+e -λ×C（t） The weight decreases to 7.59% of the original weight, with the weight decreasing to approximately 0.0759. This means that the more severe the conflict, the more significant the weight decay.

[0110] In this embodiment of the invention, by (1 / (1+e) -λ×C（t） The weights of conflicting modalities are dynamically compressed (the more severe the conflict, the greater the weight compression ratio) to reduce their impact on the fusion result, prioritizing the retention of semantically consistent modal information. In this embodiment of the invention, the fusion process from S410 to S450 is re-executed using the adjusted weights to generate new global fusion features, and the semantic conflict detection step is repeated until one of the following termination conditions is met: Condition 1: If the newly detected conflict score C(t+1)≥0 (semantic consistency), output the current global fusion feature as the corrected global fusion feature; Condition 2: When the number of iterations reaches the preset maximum threshold, output the global fusion feature of the last iteration. The preset maximum threshold can be 3 iterations. Experiments have verified that 3 iterations can cover more than 95% of conflict resolution scenarios, avoiding efficiency loss caused by excessive iterations.

[0111] S600 converts globally fused features without semantic conflicts into semantic information, and supplements fine-grained data according to the fine-grained requirements of the extraction instructions to form preliminary semantic extraction results.

[0112] The S600 specifically includes: S601 transforms semantically conflict-free globally fused features into parsable semantic signals (such as distilled Vicuna-7B) using a feature-language mapping layer. The core operations are as follows: Input preprocessing: Modal dimension separation and semantic label binding are performed on the global fusion features. Specifically, the global associated feature subsets of each modality (such as text subset, image subset, audio subset) are split according to the concatenation order of S450 (temporal modality → spatial modality), and modal type labels (such as [TEXT], [IMG], [AUDIO]) are attached to each subset to ensure that the LLM can identify the feature source; Mapping layer structure: A "bilinear projection + layer normalization" design is adopted to uniformly map each modal feature subset to the input dimension of the LLM (e.g., the input dimension d_model=4096 of Vicuna-7B): SemSignalx=LayerNorm(Wx·Fx+bx).

[0113] Where x represents the modality type, Wx is the modality-specific projection matrix (dimensions [4096, 1024]), bx is the bias term, and SemSignalx is the unimodal semantic signal.

[0114] Global semantic signal integration: The SemSignalx signals from each modality are concatenated according to "temporal logic + spatial logic" (e.g., text semantic signal → image spatial semantic signal → audio temporal semantic signal) to form the final global semantic signal SemSignal. global (Dimension [T_sem, 4096], where T_sem is the semantic sequence length), ensuring that the signal contains cross-modal temporal / spatial correlations.

[0115] S602, the global semantic signal SemSignal global Input a lightweight LLM model and guide the model to extract abstract semantics from the global fusion features using a pre-defined parsing template, specifically including: Template design: The Prompt is fixed as "Based on the following multimodal semantic signals, extract 3 types of core information: 1. Entities (including type and attributes, such as [entity: vehicle, type: vehicle, attribute: color = red]); 2. Relationships (including subject-verb-object triples, such as [vehicle-collision-guardrail]); 3. Events (including time, location, and subject, such as [event: vehicle collision, time: 8:00, location: intersection, subject: red sedan]): {SemSignal_global}"; LLM output format: The model outputs the parsed results in a structured JSON format.

[0116] Preliminary semantic extraction result definition: The structured JSON result output by LLM is denoted as Init_Sem_Result, which serves as the basic framework for subsequent fine-grained supplementation.

[0117] S603, through the parallel embedding interface of submodules, calls preset specialized submodules to synchronously supplement fine-grained data to Init_Sem_Result based on the fine-grained requirements of the extraction instructions (such as "extracting pixel coordinates of the collision area" and "extracting the timestamp of the braking sound"). The specific process is as follows: Fine-grained requirement judgment: Parse the "fine-grained keywords" in the user's extracted instructions (such as "pixel coordinates" corresponding to fine-grained visual requirements, "timestamp" corresponding to fine-grained audio requirements), and match the corresponding special sub-modules (the sub-module library has preset SEEM visual segmentation module, SHAS audio segmentation module, etc.); Submodule calling rules: If the requirement is for fine-grained visual data (such as "pixel coordinates of the collision area"): call the SEEM module, input the original image data of the S200 coding layer (including spatial location coding), and output the pixel coordinate range of the target area (such as [x1:200,y1:300,x2:400,y2:500]). If the requirement is for fine-grained audio (such as "brake sound timestamp"): call the SHAS module, input the raw audio data of the S200 encoding layer (including temporal position encoding), and output the timestamp of the target audio segment (such as [start:1.2s, end:1.5s]). The submodules run synchronously with the parallel encoding process: the input of the submodules is directly associated with the original modal data of S200 (to avoid repeated encoding), and the output results are imported into SemSignal_global in real time through "modal index alignment" (such as the spatial index of the global associated features of the image associated with the pixel coordinates output by SEEM). Fine-grained data integration: The fine-grained data output by the submodule is supplemented to the corresponding fields of Init_Sem_Result in the form of "attribute expansion".

[0118] S604: Performs format validation on Init_Sem_Result (ensuring field integrity and data type consistency), and finally outputs a structured preliminary semantic extraction result (supporting common formats such as JSON and XML), providing a foundation for subsequent semantic refinement or task applications (such as multimodal question answering and event tracing).

[0119] S700 verifies the completeness of attributes in the preliminary semantic extraction results based on the knowledge graph. If an attribute is missing, a completion process is triggered to obtain a verification result that meets the completeness standard.

[0120] Furthermore, the S700 specifically includes: S710 matches each entity in the preliminary semantic extraction results with the standard entity types in the knowledge graph to obtain the corresponding matching results.

[0121] In this embodiment of the invention, the knowledge graph is defined as a preset domain knowledge graph (such as "traffic accident multimodal knowledge graph"). Its structure includes a hierarchical relationship of "entity type - standard attribute" (example shown in Table 1 below), and each entity type is bound to a unique set of standard attributes (the attribute type includes "required attributes" and "optional attributes", and only the required attributes are included in the integrity calculation).

[0122] Table 1 Vehicles (means of transportation) Model, color, license plate, and years of use 4 Collision event (event) Time of occurrence, location of occurrence, parties involved, and outcome of the incident 4 Road guardrails (facilities) Material, location coordinates, and road section 3 In this embodiment of the invention, the matching result may include either a successfully matched standard entity type or a "pending confirmation entity" marker indicating a failed match. Specifically, for each entity in the preliminary semantic extraction result Init_Sem_Result output by S604 (such as "red sedan" or "intersection collision event"), the corresponding standard entity type is matched in the knowledge graph using the entity name and type label (such as [entity: vehicle, type: transportation]). If an entity in the preliminary result (such as "unknown object") cannot be matched with a standard type in the knowledge graph, it is marked as a "pending confirmation entity" and is not included in the attribute integrity calculation (it will be supplemented later through manual verification or a general submodule).

[0123] S720, based on the aligned standard entity type, calculate the attribute integrity representation value P for each entity in the preliminary semantic results.

[0124] Where P = Ne / Nt × 100%, Ne is the number of standard required attributes that have been successfully extracted in the preliminary semantic extraction results (the attribute values ​​must be non-empty and in a valid format, such as “color=red” being a valid extraction and “color=" being an invalid extraction, and not counted in Ne), and Nt is the total number of standard required attributes of this type of entity in the knowledge graph.

[0125] If the initial semantic extraction results contain multiple types of entities, the "overall attribute integrity representation value" is taken as the minimum value among the corresponding P values ​​of all types of entities, ensuring that the required attributes of each core entity meet the basic integrity requirements (e.g., "vehicle" P=60%, "collision event" P=100%, then the overall P=60%).

[0126] S730, if P < P0, call the preset fine-grained submodule to extract missing attributes from the multimodal data to be processed, complete them and recalculate P until P ≥ P0; if P ≥ P0, directly use the preliminary semantic extraction result as the verification result of the integrity check passing.

[0127] P0 is a preset integrity representation threshold, for example, 80%. Through domain experiments, it has been verified that this threshold can balance the requirement for attribute integrity with extraction efficiency and avoid resource waste caused by excessive completion.

[0128] Specifically, if the overall attribute integrity representation value P≥P0: the preliminary semantic extraction result Init_Sem_Result is directly used as the integrity verification result; If the overall attribute integrity representation value P < P0: (1) Identification of missing attributes: For each type of entity where P < P0, identify the missing standard required attributes (e.g., the "vehicle" entity is missing the "license plate" and "years of use" attributes). (2) Submodule mapping and calling: Establish a directional mapping relationship between "missing attributes - fine-grained submodules" (for example, when "vehicle" is missing "license plate", call the SEEM visual segmentation + OCR recognition submodule; when "collision event" is missing "occurrence time", call the SHAS audio segmentation + timestamp alignment submodule). The input source of the submodule is the original modal data of the S200 coding layer (such as original image, audio) or the global correlation features output by S450 (including spatial / temporal coding to avoid repeated calculations). (3) Attribute completion and update: Call the corresponding submodule to extract the missing attributes, add them to the preliminary semantic extraction result in the form of "attribute expansion", and update it to Sem_Result_updated; (4) Iterative verification: Based on Sem_Result_updated, repeat the calculation steps of S720 to obtain a new overall attribute integrity representation value Pnew; repeat the process of "missing item identification → submodule call → completion update → Pnew calculation" until any of the following termination conditions are met: Condition 1: Pnew ≥ P0; Condition 2: The number of iterations for completion reaches the preset maximum threshold (domain experiment verification: 2 iterations can cover more than 90% of attribute completion requirements; attributes that still cannot be completed after more than 2 iterations are mostly information that cannot be extracted from the data and do not need to be iterated further). Output results: After the termination condition is met, output the verification result of attribute integrity (Sem_Result_updated or the result of the last iteration).

[0129] S800, based on the verification result, generates a multi-granularity unified information extraction result that includes global extraction results and fine-grained grounding information.

[0130] In this embodiment of the invention, the multi-granularity unified information extraction result consists of two parts: "global extraction result" and "fine-grained grounding information". The global extraction result is the structured abstract semantic information in the verification result, including an entity list (containing entity name, type and core attributes), relation triples (subject-verb-object association), and event graph (event elements and association relationships), which is used to meet the needs of rapid application scenarios (such as multimodal summarization and event overview). The fine-grained grounding information is the original modal fine-grained data (such as pixel-level segmentation mask of entities in images, timestamp boundaries of event segments in audio, and character-level positioning coordinates of entities in text) bound to the entities / events in the global extraction result, which is used to support high-precision application scenarios (such as entity tracing and detailed verification).

[0131] The association between fine-grained grounding information and global extraction results is established through a "grounding mapping layer": for each entity or event in the global extraction results, at least one corresponding original modal fine-grained data is bound (such as binding an image segmentation mask to a "vehicle" entity and an audio timestamp to a "collision event"), and the accuracy of the association is ensured by optimizing the mapping accuracy index.

[0132] Mapping accuracy is optimized using the following formula: Lg=∑ m v=1 IoU(Mv, Bv).

[0133] Where m is the total number of association pairs between fine-grained grounding information and global results, i.e., the total number of entity / event-fine-grained data pairs that need to establish mapping relationships; v is the index of the association pair, ranging from 1 to m, used to traverse each association pair; Mv is the original modal fine-grained data in the v-th association pair (such as the segmentation mask of entities in an image, represented by a set of pixel coordinates; the time mask of event segments in audio, represented by [start, end] timestamps); Bv is the baseline boundary of the entity / event corresponding to the global extraction result in the v-th association pair (such as the character position boundary of entities in text, represented by [start_idx, end_idx]; the time boundary of event description in audio-to-text conversion, represented by [text_start, text_end] timestamps); IoU() is the intersection-union ratio of Mv and Bv, which is the pixel overlap in the image modality and the temporal overlap in the temporal modality, ranging from 0 to 1. The larger the value, the higher the matching degree between the fine-grained information and the global results.

[0134] By maximizing Lg, which is to maximize the sum of the cross-union ratios of all associated pairs, the mapping accuracy between fine-grained grounding information and global extraction results is improved, and finally, a multi-granularity unified information extraction result with accurate association and complete hierarchy is output.

[0135] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in this invention.

[0136] This invention also provides a computer-readable storage medium storing computer-executable instructions for performing the methods described in this invention.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for dynamic adaptation and collaborative extraction of large models based on a unified framework, characterized in that, The method includes the following steps: S100 performs dynamic modality recognition and adaptation on the multimodal data to be processed, and encodes the features of each modality to obtain the basic features of each modality. S200 identifies the semantic anchor points of each modal data, concatenates the basic features and semantic anchor points of each modal data, and inputs them into the corresponding parallel Transformer encoder for encoding to obtain the fused features of each modal data; at the same time, a monitoring node is set at the output of each layer of each encoder to monitor and correct feature distortion in real time. S300, based on the fusion features of data from each modality, realizes cross-modal information interaction and outputs cross-modal feature vectors of each modality; S400: Obtain the relationship type between each modal data, then determine the corresponding modal fusion weight based on the obtained relationship type, and perform differential fusion on the cross-modal feature vectors of each modal data to obtain the current global fusion feature; S500: Detect the semantic consistency of the global fusion features; if semantic conflicts exist, dynamically adjust the current fusion weights of each modality and return to S400; otherwise, execute S600. S600 converts globally fused features without semantic conflicts into semantic information, and supplements fine-grained data according to the fine-grained requirements of the extraction instructions to form preliminary semantic extraction results; S700 verifies the completeness of attributes in the preliminary semantic extraction results based on the knowledge graph. If an attribute is missing, a completion process is triggered to obtain a verification result that meets the completeness standard. S800, based on the verification result, generate a multi-granularity unified information extraction result that includes global extraction results and fine-grained grounding information; S100 specifically includes: S110, based on modal feature entropy values ​​and a preset multimodal embedding model, automatically identify the modal type of the multimodal data to be processed. The modal type includes known modalities and unknown modalities. Known modalities refer to modalities natively supported by the ImageBind model, including text, images, audio, video, radar point clouds, and depth maps. Unknown modalities refer to novel information carriers not natively supported by the ImageBind model, including time-series electrical signals from novel sensors, text strings with novel encoding formats, and biometric signals. S120, feature encoding and unified embedding processing are performed on the identified known modalities and unknown modalities respectively to obtain the initial features of the known modalities and unknown modalities respectively; among them, for unknown modalities, the corresponding feature extraction submodule is triggered according to the physical properties of the unknown modalities, and after performing preliminary feature extraction, it is mapped to the unified embedding space of the preset multimodal embedding model to form the initial features of the unknown modalities. S130, dynamically adjust the initial features of known and unknown modalities to obtain a standard tensor in a preset format, wherein the standard tensor is the basic feature of each modal data; In S120, if the unknown mode is a continuous signal, the feature extraction submodule automatically uses Fourier transform and wavelet analysis tools to extract frequency domain and time domain features; if the unknown mode is a discrete structure, the feature extraction submodule captures local structural features through character-level embedding and n-gram statistical methods.

2. The method according to claim 1, characterized in that, In the S200, a monitoring node is set at the output of each layer of each encoder to monitor and correct feature distortions in real time, specifically including: For each encoder, the normal feature vector of the current frame feature vector is determined according to its corresponding coding mode type. If the coding mode of the encoder is a temporal mode, the normal feature vector of the current frame is calculated using the sliding window method. If it is a non-temporal mode, the feature mean of the same type of actual data in the application scenario is used as the normal feature vector of the current frame. Calculate the difference between the current frame feature vector and the normal feature vector at the current encoder output. If the difference is greater than a preset threshold, it is determined that the current frame feature vector has feature distortion. If feature distortion is detected, an adaptive error correction mechanism is immediately triggered. If the difference value of the feature vector of the current frame is still greater than the preset threshold after the error correction mechanism is triggered m times consecutively, the multimodal data to be processed corresponding to the encoder is marked as abnormal data and temporarily stored in the abnormal database. The subsequent processing of the abnormal data is suspended, and a decision is made on whether to restart the processing after the data quality is manually reviewed and confirmed.

3. The method according to claim 1, characterized in that, The S400 specifically includes: S410, obtain the sharing score S(M1, M2) between any two modalities. If S(M1, M2) ≥ S0, determine that the two modalities are in a sharing relationship; otherwise, determine that they are in an independent relationship. Wherein, S0 is a preset sharing threshold; M1 is one of the two modalities, and M2 is the other of the two modalities. S420, For two modalities determined to have a sharing relationship, the following steps are used to determine the weights of M1 absorbing information from M2 and the weights of M2 absorbing information from M1, which are then used to generate the global association features for each of the two modalities: S4201, based on the semantic attributes of the core related entities that both modalities point to, select directly related feature subsets from the cross-modal feature vector V1 of M1 and the cross-modal feature vector V2 of M2, remove redundant noise features, and obtain the selected features VF1 and VF2. S4202 uses a cross-attention mechanism to calculate the correlation between feature pairs in VF1 and VF2; S4203, based on the correlation between feature pairs in VF1 and VF2, assigns initial weights W1 to the two modalities to absorb information from each other. init and W2 init ; S4204, with the objective of minimizing the modal complementarity loss function, adjusts the initial weights in reverse to obtain the optimized fusion weights W1. opt and W2 opt ; S4205, based on W1 opt and W2 opt Update the features of M1 and M2 respectively; S430, for two modalities M1 and M2 that are determined to be independent, a weak fusion strategy is adopted to avoid interference from irrelevant information and update the correlation features of the two modalities: V1 and V2 remain in independent encoding states and do not engage in cross-modal feature depth interaction; Both the weight of M1 absorbing information from M2 and the weight of M2 absorbing information from M1 decrease according to the following formula: W weak =W0×(1-S(M1, M2)) is calculated, where W weak W0 is the base weight for modal fusion; In the weight vector, the elements that are cross-correlated with the two independent modes are set to 0 to prevent contamination by irrelevant features; Based on W weak Update the features of M1 and M2 respectively; S440, for each modality Mx, generate global association features of Mx by interacting with and cumulatively updating all other modalities My in the multimodal data to be processed; S450 concatenates the global associated features of all modalities in a preset order to obtain concatenated features, and then unifies the feature distribution of the concatenated features through a linear transformation layer, finally outputting the current global fusion features.

4. The method according to claim 3, characterized in that, The modal complementarity loss function satisfies the following condition: L comp =∑ n i=1 |f1(i)-f2(i)|•I(f1(i)≠0∨f2(i)≠0), where f1(i) is the i-th element of the feature vector of one modality in the shared relation after processing by S4201, f2(i) is the i-th element of the feature vector of another modality in the shared relation after processing by S4201, n is the dimension of the feature vector, I() is the indicator function, ∨ is the logical OR operator, where if f1(i)≠0 and / or f2(i)≠0, then I()=1, if f1(i)=0 and f2(i)=0, then I()=0, and • represents dot product.

5. The method according to claim 1, characterized in that, In S500, if a semantic conflict is detected in the global fusion features, the fusion weights of each modality are adjusted as follows: Wt d =Wt×(1 / (1+e -λ×C（t） Wt represents the modal fusion weights at time t. d Let e ​​be the adjusted modal fusion weight at the current time t, λ be the natural constant, λ be the adjustment coefficient, and C(t) be the semantic conflict score at the current time t.

6. The method according to claim 1, characterized in that, The S700 specifically includes: S710, Match each entity in the preliminary semantic extraction results with the standard entity types in the knowledge graph to obtain the corresponding matching results; S720, based on the aligned standard entity type, calculate the attribute integrity representation value P for each entity in the preliminary semantic results; S730, if P < P0, call the preset fine-grained submodule to extract missing attributes from the multimodal data to be processed, complete them and recalculate P until P ≥ P0; if P ≥ P0, directly use the preliminary semantic extraction result as the verification result of the integrity verification, where P0 is the preset integrity characterization value threshold.

Images