Large model dynamic adaptation and collaborative extraction method based on unified framework

Abstract

The invention provides a large model dynamic adaptation and collaborative extraction method based on a unified framework. The method comprises the steps of preprocessing to-be-processed multi-modal data; dynamically identifying a modal type and carrying out adaptive coding to obtain a basic feature; the semantic anchor points are combined to generate fusion features through a parallel Transform encoder, feature distortion is synchronously monitored, and error correction is carried out; realizing cross-modal information interaction based on a cross-attention mechanism of an anchor point weight enhancement matrix, and generating a cross-modal feature vector; determining a differential fusion weight according to the type of the relationship between the modes, and obtaining a global fusion feature; semantic consistency is detected, and the weight is dynamically adjusted; semantic information is converted, fine-grained data is supplemented, and a preliminary result is formed; verifying and complementing attributes based on the knowledge graph; and generating a multi-granularity extraction result containing the global result and the fine-granularity grounding information. According to the method, efficient adaptation, accurate fusion and high-quality information extraction of multi-modal data can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal information processing, in particular to a large model dynamic adaptation and collaborative extraction method based on a unified framework. BACKGROUND

[0002] With the rapid development of information technology, technological innovations represented by artificial intelligence, 5G communication and cloud computing are driving multi-modal data to grow at an unprecedented speed and scale. Multi-modal data covers various information carriers such as text, images, audio and video. Typical forms include images with text annotations and videos with voice explanations. Such data is growing explosively in the fields of the Internet, intelligent monitoring, autonomous driving, medical diagnosis and educational technology. In the field of the Internet, social media platforms generate massive multi-modal content combining images, videos and comments every day; intelligent monitoring systems need to process camera images and environmental sounds simultaneously; in the autonomous driving scenario, vehicles need to analyze sensor images, radar data and navigation voice instructions in real time. These application scenarios have higher requirements for multi-modal information extraction technology.

[0003] However, the current development of multi-modal information extraction technology is significantly lagging behind the application demand, and there are significant technical bottlenecks, which are embodied in the following key dimensions: Modality adaptability limitation: Most traditional multi-modal information extraction methods are designed for fixed modality combinations, such as the common "text + image" mode. When facing dynamic modality inputs, such as the complex combination of "text + audio + video", the system needs to reconfigure the encoding logic. This not only increases the development cost and time, but also leads to extremely low adaptation efficiency, making it difficult to meet the needs of application scenarios with high real-time requirements.

[0004] Insufficient semantic association mining: Existing technologies in multi-modal feature fusion mostly stay at the surface feature level and are difficult to deeply mine the deep semantic associations between different modalities. For example, in the traffic accident analysis scenario, there is a close implicit association between the "collision intensity" described in the text and the deformation area of the vehicle in the image and the collision decibel value recorded in the audio. However, traditional methods often cannot effectively capture these associated information, resulting in incomplete information extraction.

[0005] Lack of error robustness: In the information extraction process, existing technologies are prone to problems such as content duplication, incomplete information, and incorrect matching. More seriously, the system generally lacks real-time monitoring and correction mechanisms, and once an error occurs, it will continue to propagate in the subsequent processing process, eventually leading to a serious deviation of the extraction result from the actual situation.

[0006] Modal relationship processing is rigid: there are various relationships between different modalities, such as "sharing" and "independence". When text and image description describe the same entity, they belong to the "sharing" relationship; and when text and audio content are irrelevant, they present the "independence" relationship. However, existing technologies often use a unified fusion strategy to process all modal relationships, which is easy to introduce noise or lose key information, affecting the accuracy of information extraction.

[0007] Therefore, in order to meet the increasingly complex application requirements, it is urgent to develop a multi-modal unified information extraction technology that can adaptively process dynamic modal input, strengthen deep semantic association, correct errors in real time, and dynamically adapt modal relationships. The development of this technology not only breaks through the bottleneck of existing technologies, but also provides strong support for the efficient use of multi-modal data, and promotes the development and application innovation of related fields. SUMMARY

[0008] To solve the above technical problems, the technical solution adopted by the present application is: The embodiment of the present application provides a large model dynamic adaptation and collaborative extraction method based on a unified framework, which comprises the following steps: S100, dynamic modal recognition and adaptation are performed on the multi-modal data to be processed, and the feature encoding of each modal data is performed to obtain the basic features of each modal data.

[0009] S200, the semantic anchor points of each modal data are identified, the basic features and semantic anchor points of each modal data are spliced and input into the corresponding parallel Transformer encoder for encoding to obtain the fusion features of each modal data; at the same time, a monitoring node is arranged at the output end of each encoder to monitor and correct the feature distortion in real time.

[0010] S300, cross-modal information interaction is realized based on the fusion features of each modal data, and the cross-modal feature vectors of each modality are output; S400, the relationship types between each modal data are obtained, and the corresponding modal fusion weights are determined based on the obtained relationship types, and the cross-modal feature vectors of each modal data are differentially fused to obtain the current global fusion features.

[0011] S500, the semantic consistency of the global fusion features is detected; if there is a semantic conflict, the current modal fusion weights are dynamically adjusted and returned to S400, otherwise S600 is executed.

[0012] S600, the global fusion features without semantic conflict are converted into semantic information, and according to the fine-grained requirements of the extraction instruction, fine-grained data is supplemented to form the preliminary semantic extraction result.

[0013] S700, based on the knowledge graph, the attribute integrity of the preliminary semantic extraction result is verified, if the attribute is missing, the completion process is triggered, and the verification result with complete integrity is obtained.

[0014] S800, based on the verification result, a multi-granularity unified information extraction result containing global extraction result and fine-grained grounding information is generated.

[0015] The present application has at least the following beneficial effects: (1) Breakthrough modal adaptation bottleneck, significantly improve multi-modal compatibility Through dynamic modal recognition and adaptive design, based on modal feature entropy value and ImageBind model, known modal (text / image / audio / video) and unknown modal (such as new sensor time series signal, special coded text) can be automatically identified, and the feature dimension and coding strategy are dynamically adjusted according to the physical properties of different modalities (such as Fourier transform for continuous signal to extract frequency domain features, character level embedding for discrete structure to capture local features). Compared with the existing fixed modal adaptation scheme, this method can be compatible with at least 3 kinds of unknown modalities, and the modal adaptation coverage is increased by more than 60%, solving the technical pain points of "unable to identify and difficult to code" for unknown modalities, meeting the processing needs of multi-source heterogeneous multi-modal data in complex scenarios.

[0016] (2) Block feature distortion propagation and ensure feature processing quality stability A monitoring node is arranged at the output end of each layer of the parallel Transformer encoder, and the normal feature vector is predicted by the sliding window method (time series modal) or the mean value of similar data (non-time series modal), and the feature distortion is detected by the difference formula, and the re-encoding mechanism is triggered (abnormal for 3 times in a row). The design can intercept feature distortion caused by device noise and transmission packet loss in real time, the feature distortion correction rate is ≥95%, avoiding the distortion feature into the subsequent fusion process, so that the accuracy of cross-modal feature interaction is improved by more than 40%, solving the problem of "feature error hidden propagation and difficult to correct" in traditional methods, and ensuring the stability of the feature quality of the whole process.

[0017] (3) Realize differentiated cross-modal fusion and improve semantic fusion accuracy Based on the modal sharing degree score, the shared / independent relationship is distinguished, the shared relationship is deeply bound to the high correlation feature through the cross attention mechanism and combined with the modal complementary loss function to optimize the weight, and the independent relationship adopts weak fusion + modal isolation mask to prevent interference. Compared with the traditional "uniform weight fusion" scheme, the semantic correlation capture accuracy of the shared relationship modal is improved by 50%, and the information interference of the independent relationship modal is reduced by 70%, effectively solving the fusion rigid problem of "insufficient fusion of shared modal information and cross pollution of independent modal", so that the semantic consistency of the global fusion feature is ≥90%. The (4) Recheck to ensure result quality, improve information extraction completeness and accuracy Through lightweight LLM detection of semantic conflicts (time / space / attribute contradictions) of global fusion features, dynamically adjust weight iteration optimization (conflict correction rate ≥ 85%); based on knowledge graph to check attribute integrity (threshold, trigger sub-module to complete missing attributes (such as OCR extracted text attributes, SEEM segmented visual attributes). Double checking makes the attribute integrity rate of the preliminary semantic extraction result from 60% of the traditional method to more than 92%, and the semantic conflict rate is reduced to less than 5%, solving the problem of "semantic contradiction of extraction result and missing key attributes" of traditional methods, ensuring the availability and reliability of output information.

[0018] (5) Output multi-granularity results to adapt to the needs of diversified application scenarios Generate "global extraction results (entity list / relationship triplets) + fine-grained grounding information (segmentation mask and entity binding relationship)" multi-granularity results, global results meet the "quick decision" scene needs of intelligent monitoring and early warning, e-commerce rapid retrieval (response speed improved by 30%), fine-grained information meets the "accurate analysis" scene needs of accident responsibility identification, medical lesion traceability (detail information coverage rate improved by 80%). Compared with the traditional single format output scheme, this method can adapt to the information use needs of at least 5 different application scenarios, expanding the application boundary of multi-modal information extraction technology.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 The flowchart of the method for dynamic adaptation and collaborative extraction of large models based on a unified framework provided by the embodiments of the present application. DETAILED DESCRIPTION

[0022] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of the present application.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0024] It is noted that some of the example embodiments are described as a process or method depicted as a flowchart. Although the flowchart describes the steps as occurring in a sequence, many of the steps can be performed concurrently, in parallel, or simultaneously. In addition, the order of the steps can be re-arranged. A process can terminate when its operations are completed, but could also terminate without reaching an end, for example, if it is interrupted, or if the pertinent data is no longer available. A process can correspond to a method, function, routine, subroutine, or the like.

[0025] The present application aims to solve the technical problems of poor modal adaptability, weak semantic association, low error robustness, and rigid modal relationship processing in existing multi-modal information extraction technology. By constructing an innovative large model dynamic adaptation and collaborative extraction method and system based on a unified framework, the barriers between modalities are broken down, the semantic association between different modal data is enhanced, the robustness of the model in complex data environment is improved, the modal relationship is flexibly processed, and efficient and accurate information extraction of multi-modal data is realized, meeting the urgent needs of various industries for deep mining and utilization of multi-modal information.

[0026] The embodiment of the present application provides a large model dynamic adaptation and collaborative extraction method based on a unified framework, as shown in Figure 1 The method can include the following steps: S100, dynamic modal recognition and adaptation are performed on the multi-modal data to be processed, and feature encoding is performed on the modal data to obtain the basic features of the modal data.

[0027] Before S100, the following steps are further included: Step 1: Obtain original multi-modal data and extraction instructions.

[0028] In this embodiment of the invention, multimodal data refers to a data set containing two or more heterogeneous information carriers. These information carriers have different physical properties and forms of expression, including but not limited to: text (such as natural language descriptions, structured forms), images (such as still pictures, remote sensing images), audio (such as voice clips, environmental sound effects), video (such as continuous frame sequences, surveillance videos), radar point clouds (such as three-dimensional spatial coordinate data), time-series signals (such as electrical signals collected by sensors, temperature change curves), etc.

[0029] Extraction instructions are structured / semi-structured instructions initiated by users, downstream business systems (such as intelligent analysis platforms and automated decision-making systems), or application scenarios to specify the objectives, scope, accuracy, and output requirements for multimodal information extraction. They serve as the core basis for subsequent multimodal data processing (such as dynamic modality recognition and feature encoding) and semantic extraction (such as fine-grained information supplementation and global result generation), aiming to ensure that the extraction process accurately matches the actual application needs. Their key components typically include, but are not limited to: Extraction Target: Identify the core information types to be obtained from multimodal data, such as entities (e.g., "product name" in text, "vehicle target" in images), relationships (e.g., "employee-department" association, "segmentation region-entity description" binding relationship between images and text), events (e.g., event graph of "accident time-location-involved parties"), or specific attributes (e.g., "speaker identity" in audio, "target size" in radar point clouds); Fine-grained requirements: Define the granularity standard for information extraction, such as whether text needs to be extracted to the "phrase level" or "sentence level", whether image segmentation needs to be accurate to the "pixel-level mask" or "target bounding box", and whether time-series signals need to be extracted to the "real-time instantaneous value" or "time period statistical value". Modal association requirements: Specify whether cross-modal association extraction is required (e.g., "bind pedestrians in video frames to speakers in synchronized audio"), or extraction is only for a single modality (e.g., "extract building coordinates only from remote sensing imagery"). Output specifications: Define the format of the extracted results (such as structured tables, knowledge graphs, JSON strings), storage format (such as tensors, text files), and delivery method (such as real-time push, batch export). Accuracy threshold: Set the quality standard for the extraction results, such as entity recognition accuracy ≥95%, cross-modal information matching IoU (Intersection over Union) ≥0.8, etc., which are used as the basis for judgment in subsequent verification stages (such as S700 attribute integrity verification).

[0030] Step 2: Preprocess the original multimodal data to obtain preprocessed multimodal data, which will be used as the multimodal data to be processed.

[0031] In the embodiment of the application, the preprocessing includes: cleaning data noise by using an outlier detection algorithm (such as Z-score method), removing low-quality samples by sample quality score (combining definition, integrity, and annotation accuracy), correcting wrong labels based on artificial verification and rule engine, and finally converting into a computable tensor form according to the modal type, such as converting text into character index tensor and converting image into pixel value tensor.

[0032] Further, S100 specifically includes: S110, automatically identifying the modal type of the to-be-processed multi-modal data based on the modal feature entropy value and a preset multi-modal embedding model.

[0033] The preset multi-modal embedding model can be an ImageBind model, which is a multi-modal basic model proposed by Meta Company. The core function of the model is to map the features of multiple known modalities such as text, image, audio, video, depth map, and heat map to a unified embedding space (i.e., a vector space with the same feature vector dimension) through a preset modal alignment mechanism, so as to realize semantic comparability of cross-modal features. The model realizes semantic comparability of cross-modal features through a shared feature coding framework and cross-modal contrast learning, so that the content related in different modalities has a similar vector representation in the embedding space, thereby providing a basis for unified processing of multi-modal data.

[0034] The modal type includes a known modality and an unknown modality. The known modality refers to a modality type originally supported by the ImageBind model, including text, image, audio, video, radar point cloud, and depth map. The unknown modality refers to a new information carrier that is not originally supported by the ImageBind model, including but not limited to a time-series electrical signal of a new sensor (such as a millimeter wave radar derived signal), a text string of a new encoding format (such as an encrypted text of a self-defined protocol), a biological feature signal (such as a heart rate waveform), and the like.

[0035] In the embodiment of the application, the modal feature entropy value H(M) is used to quantify the complexity of the modal feature distribution, and the calculation formula is: H(M)=∑ q j=1 (p j ×log2p j ).

[0036] Wherein, M is the modal data to be identified, p j is the occurrence probability of the jth feature in the modality (such as the proportion of “high frequency energy” and “low frequency energy” in the audio modality), and q is the total number of feature categories in the modality. When H(M) is greater than H0, it is determined as an unknown modality (the feature distribution is complex, and the difference with the known modality is significant); when H(M)≤1.2, the specific known modality type is matched and determined by comparing with the known modality feature library of the ImageBind model. H0 is a preset entropy threshold value, which can be 1.2.

[0037] S120, respectively, the identified known modalities and unknown modalities are respectively subjected to feature encoding and unified embedding processing, and initial features of the known modalities and the unknown modalities are obtained.

[0038] Among them, for the known modalities: first, the physical properties of each modality (such as the semantic sequence of text, the spatial structure of image, the time sequence fluctuation of audio, etc.) are extracted by using the modal-specific encoder adapted to the modality; then, the different dimension-specific features are uniformly mapped to a preset dimension, for example, 1024-dimensional shared embedding space, through the modality adapter of the ImageBind model, to form initial features with comparable semantics, laying a foundation for subsequent cross-modal interaction.

[0039] Specifically, for the text modality: a pre-trained language model (such as BERT-base, RoBERTa) is used to encode the text sequence (such as a natural language sentence, a structured form field), and the semantic association between words is captured through the context attention mechanism, outputting a 768-dimensional word embedding vector sequence (each vector corresponds to a word or subword in the text); the 768-dimensional word embedding vector is input into the text adapter of the ImageBind model, and is converted into a 1024-dimensional unified embedding feature through linear transformation and feature alignment, preserving the semantic logical relationship of the text.

[0040] For the image modality: a convolutional neural network (such as ResNet-50, ViT) is used to extract features from static images (such as photos, remote sensing images), and the local texture, global structure and semantic information (such as object contour, scene type) of the image are captured through multi-layer convolution / self-attention operation, outputting a 2048-dimensional global visual feature vector; the 2048-dimensional visual feature is input into the image adapter of the ImageBind model, and is mapped to a 1024-dimensional unified embedding space through nonlinear transformation, ensuring that the image features of the same object / scene and the corresponding text description features are adjacent in the embedding space.

[0041] For the audio modality: first, the audio signal (such as speech, environmental sound effect) is preprocessed (sampling rate is unified to 16kHz, time length is truncated / completed to 3 seconds), and the Mel spectrogram (dimension 128x128) is generated through short-time Fourier transform; then, an audio convolutional network (such as PANNs, AudioCNN) is used to encode the Mel spectrogram, and 128-dimensional time-spectrum joint features (containing pitch, loudness, rhythm, etc.) are extracted; then, the 128-dimensional audio features are input into the audio adapter of the ImageBind model, and are converted into 1024-dimensional unified embedding features through dimension expansion and semantic alignment, realizing cross-modal association of "sound-semantics" (such as "car horn" audio features and "car" text features are adjacent).

[0042] For video modality: adopt key frame screening mechanism (such as K-means clustering based on inter-frame difference degree) to extract 16 key frames (covering the core content of the video) from the video sequence (such as surveillance video, short video); reuse the ResNet-50 encoder of the image modality to extract 2048-dimensional visual features for each key frame, and then capture the inter-frame temporal correlation through a temporal attention network (such as BiLSTM) to output 2048-dimensional video global features; input the 2048-dimensional video features into the video adapter of the ImageBind model, and convert them into 1024-dimensional unified embedding features by combining and weighting the temporal information, so as to ensure that the dynamic content (such as "vehicle driving") of the video and the corresponding text description are semantically consistent in the embedding space.

[0043] For radar point cloud modality: pre-process the radar point cloud data (such as the three-dimensional space point set collected by laser radar) (denoising, down-sampling to 1024 points), encode the three-dimensional coordinates (x, y, z) and reflection intensity features through a point cloud convolution network (such as PointNet++, PointTransformer), aggregate local neighborhood and pool global features, and output 1024-dimensional spatial structure features (containing object shape, position relationship, etc.); input the 1024-dimensional point cloud features into the point cloud adapter of the ImageBind model, align the spatial features and semantic features, maintain the 1024-dimensional unified embedding dimension, and realize the cross-modal correlation between "three-dimensional structure" and "semantic description" (such as "obstacle point cloud" and "obstacle" text features are adjacent).

[0044] For depth map modality: pre-process the depth map (single-channel image recording pixel distance information) (depth value normalization, size unified to 224x224), extract depth gradient, object contour, etc. through a depth-aware convolution network (such as DenseDepth, DepthNet), and output a 512-dimensional depth feature vector by fusing local depth difference and global scene structure; input the 512-dimensional depth feature into the depth map adapter of the ImageBind model, expand the dimension (supplement semantic information through residual connection) to convert it into 1024-dimensional unified embedding features, so that the depth information (such as "object height", "distance") and the corresponding text / image features are semantically matched in the embedding space.

[0045] For unknown modality: trigger the corresponding feature extraction submodule according to the physical properties of the unknown modality, perform preliminary feature extraction, and then map it to the unified embedding space of the ImageBind model to form the initial features of the unknown modality.

[0046] Among them, the unknown modality can be determined to be a continuous signal type or a discrete structure type by data format analysis such as whether it contains a timestamp and whether it is a continuous numerical sequence.

[0047] Wherein, the corresponding feature extraction submodule is triggered according to the physical properties of the unknown modal, and after preliminary feature extraction, it is mapped to the unified embedding space of the preset multi-modal embedding model to form the initial features of the unknown modal, which specifically includes: If the unknown modal is a continuous signal such as a new type of sensor time series electrical signal, the sub-module will automatically enable Fourier transform to extract frequency domain features, wavelet analysis to extract time-frequency domain joint features, and output a 512-dimensional feature vector; If the unknown modal is a discrete structure such as a new type of encoded format text string, then capture local structure features through character-level embedding (CharCNN), n-gram (n=2,3) statistics, etc. Method to output a 512-dimensional feature vector; Through the training of the adaptive layer (with the known modal embedding as supervision), the 512-dimensional features are mapped to the 1024-dimensional unified embedding space of the ImageBind model to form the initial features of the unknown modal.

[0048] This design does not need to preset the modal type, but only through the physical properties of the data (such as whether it is a time series signal, whether it contains spatial coordinates, etc.) to trigger the corresponding processing logic, thereby realizing the preliminary feature extraction of the unknown modal without distinction.

[0049] S130, dynamically adjust the initial features of the known modal and the unknown modal to obtain a standard tensor of a preset format, which is the basic features of each modal data.

[0050] Specifically, for the modal lacking time sequence dimension (such as static image, text), the time sequence dimension is supplemented by feature replication (copying the feature vector to the preset time sequence length); for the modal lacking spatial dimension (such as audio, time series signal), virtual spatial coordinates are generated by interpolation (spatial topology is constructed based on feature similarity); for dimension redundant modal such as long video (length>30s), high resolution image (resolution>4K), self-attention mechanism (such as TransformerEncoder) is used to focus on key feature areas, while retaining core information and compressing to the preset dimension; Finally, the output format of the standard tensor is unified, and its dimension is [batch_size, feature_dim=1024, Sequence_Len=32, Spatial_H=16, Spatial_W=16], which ensures that it can be directly input into the subsequent parallel Transformer encoder for processing.

[0051] S200, identify the semantic anchor points of each modal data, and input the basic features and semantic anchor points of each modal data into the corresponding parallel Transformer encoder for encoding to obtain the fusion features of each modal data; at the same time, a monitoring node is set at the output end of each encoder to monitor and correct the feature distortion in real time.

[0052] In the embodiments of the present application, the semantic anchor points refer to key units in each modality data carrying core semantic information (such as entities in text, core objects in images, landmark events in audio, etc.), which serve as the "semantic reference" for cross-modal information interaction, ensuring the alignment of features of different modalities in the same semantic dimension.

[0053] The identification and feature extraction of each modality semantic anchor point follow the logic of "modality-specific identifier + unified dimension embedding", which is as follows: Text modality: adopt entity recognition model (such as spaCy, BERT-NER) to identify core entities (such as names, event names, time, place, etc.) and semantic keywords (such as verbs, adjectives) in text as semantic anchor points; convert the anchor points into 768-dimensional embedding vectors through pre-trained language models (such as BERT), preserving the semantic attributes of the anchor points (such as entity type, semantic role).

[0054] Image modality: adopt target detection model (such as YOLOv8, FasterR-CNN) to identify key regions (such as main objects, scene markers) in images as semantic anchor points; encode the anchor point regions (image sub-regions cropped by bounding boxes) through visual Transformer (such as ViT) to output 768-dimensional visual anchor point embedding vectors, containing features such as shape, color, and position of the anchor points.

[0055] Audio modality: adopt audio event detection model (such as PANNs, YAMNet) to identify feature events (such as voice instructions, specific sound effects, environment landmark sounds) in audio as semantic anchor points; perform Mel spectrum analysis on the audio segments (such as 1-3 second voice segments) corresponding to the anchor points, extract 128-dimensional time sequence features through audio Transformer, and then map them into 768-dimensional audio anchor point embedding vectors.

[0056] Video modality: combine temporal anchor points and spatial anchor points - identify key action segments (such as "vehicle turning" and "person raising hand") in video as temporal anchor points through temporal action detection model (such as SlowFast), and identify core objects in the segments as spatial anchor points through target detection model; after fusing temporal features and spatial features, convert them into 768-dimensional video anchor point embedding vectors.

[0057] Radar point cloud modality: adopt point cloud segmentation model (such as PointRCNN) to identify key targets (such as pedestrians, vehicles, obstacles) in point cloud as semantic anchor points; aggregate features of the point sets (sub-point clouds extracted by clustering) corresponding to the anchor points to output 768-dimensional point cloud anchor point embedding vectors, containing information such as three-dimensional structure and motion state of the target.

[0058] Depth map modality: adopt a depth feature segmentation model (such as Depth-Aware CNN) to identify key depth regions (such as "nearby objects" and "far scene boundaries") in the depth map as semantic anchors; extract features such as depth gradient and distance distribution of the anchor region, and convert them into 768-dimensional depth anchor embedding vectors.

[0059] In the embodiment of the application, the parallel Transformer encoder is a "multi-branch independent encoding + cross-modal interaction fusion" architecture, which includes an input layer, an encoding layer, and an interaction layer. Each modality corresponds to an independent encoder branch, and the branches are connected through the interaction layer to realize information association. Specifically, the following is provided: (1) Input layer: feature concatenation and dimension alignment The "basic features" (standard tensor output by S100, dimension [batch_size, 1024, 32, 16, 16]) and "semantic anchor features" (768-dimensional embedding vector) of each modality are processed as follows: Dimension alignment: the 768-dimensional anchor embedding vector is expanded to 1024-dimensional through linear mapping, matching the feature dimension of the basic features; Feature concatenation: the aligned anchor features and the basic features are concatenated according to the channel dimension to generate the final input tensor of each modality, with a dimension of [batch_size, 2048, 32, 16, 16] (basic features 1024-dimensional + anchor features 1024-dimensional), ensuring that the encoder receives both global features and core semantic anchor information.

[0060] (2) Encoding layer: modality-independent deep encoding Each modality encoder branch operates independently, taking the concatenated tensor of "basic features + semantic anchor features" as input (dimension [batch_size, 2048, 32, 16, 16]), and realizing the organic fusion of the two parts of information through deep encoding, finally outputting the fusion features of each modality. A single branch contains N=6 layers of encoder blocks (extensible to 12 layers), and the structure and function of each layer of encoder block are designed around "feature fusion", as follows: Position encoding: Injecting time / space position information into the concatenated features, covering the position correlation between base features and semantic anchor features - for time-series modalities such as text / audio / video, simultaneously labeling the time-series position of base features (e.g., text sequence position, audio frame order) and the time-series coordinates of semantic anchors (e.g., entity word position in text, event timestamp in audio) in the time dimension; for spatial modalities such as images / point clouds / depth maps, synchronously encoding the spatial distribution of base features (e.g., pixel coordinates, point cloud three-dimensional position) and the spatial boundaries of semantic anchors (e.g., object detection box coordinates, key region outline) in the spatial dimension, ensuring that the model captures both the "position correlation of base features themselves" and the "position correspondence between base features and anchor features". Self-attention mechanism: Calculate the correlation weight within the feature through Query (Q), Key (K), and Value (V) (e.g., the correlation between "event" and "time" in text, the correlation between "object" and "background" in image), the formula is: Self-Attention (Q, K, V) = softmax (QK T / (d k ) 1 / 2 ) V.

[0061] where d k is the Q / K vector dimension (value 512), focusing on the key features within the modalities through self-attention. Q, K, and V come from the concatenated tensor of "base features + semantic anchor features", and the attention calculation not only focuses on the key features within the modalities (e.g., the correlation between "event" and "time" in text, the correlation between "object" and "background" in image), but also captures the binding relationship between "base features and semantic anchors" (e.g., the correlation between "accident" description in text base features and "collision" entity anchor, the correlation between "fuzzy area" in image base features and "vehicle" target anchor), and through weight distribution, strengthens the semantic constraints of anchors on base features.

[0062] Feedforward neural network: composed of two linear transformations (intermediate dimension set to 2048) and ReLU activation function, for nonlinear mapping of "base feature-anchor feature" correlation features output by self-attention, further integrating the deep semantics of the two parts of information (e.g., converting the correlation information between "vehicle outline" base features and "accident vehicle" anchor features in image into more abstract "accident subject" semantic features), enhancing the expression ability of the fused features.

[0063] Residual connection and layer normalization: Add residual connection after each step of operation (add input features to current layer output features) to avoid information loss in the fusion process of "basic features-anchor features"; at the same time, stabilize the feature distribution through layer normalization to ensure that after 6 layers of encoding, the output feature tensor has deep fusion of basic features and semantic anchor features, forming a modal fusion feature with global information and core semantic benchmark (dimension remains [batch_size, 2048, 32, 16, 16]), laying a foundation for subsequent cross-modal interaction of the interaction layer.

[0064] (3) Interaction layer: cross-modal anchor association enhancement The interaction layer includes a cross-attention information fusion module, which strengthens the association between different modal semantic anchors through an "anchor weight enhancement matrix" to achieve cross-modal deep semantic alignment. The specific mechanism is as follows: Cross-attention calculation: For any two modalities (such as text and image), take the features of modality A as Query (Q), and the semantic anchor features of modality B as Key (K) and Value (V), and output the directional attention results of modality A to modality B through the cross-modal attention calculation mechanism, where the cross-modal attention calculation mechanism is: Output AB =softmax((QK T ⊙W a ) / (d k ) 1 / 2 )・V。

[0065] Where: Output AB is the one-way cross-attention result of modality A to modality B, W a is an anchor weight enhancement matrix (dimension same as K), with diagonal elements taking values 1.2-1.3 (strengthening the weight of the semantic anchor itself), and non-diagonal elements taking values 1 (maintaining the basic weight of non-anchor features), which amplifies the role of anchors in cross-modal association (such as the association between the text "car" entity and the image "car" target) through the matrix; ⊙ represents element-wise multiplication, and "・" represents matrix multiplication. The normalized attention weight is used to weight V, and the cross-modal feature vector of modality A with the same dimension as Q is output.

[0066] Multi-modal parallel interaction: all modal branches interact with each other through the above cross-attention mechanism (such as text ↔ image, audio ↔ video, etc.), and finally output cross-modal feature vectors of each modality data fused with cross-modal association information.

[0067] Further, in S200, a monitoring node is set at the output end of each layer of the encoder, and through the closed-loop mechanism of "normal feature benchmark construction → real-time difference detection → hierarchical error correction and abnormal handling", the feature distortion is accurately intercepted and corrected, which specifically includes: S210, for each encoder, according to its corresponding encoding modal type, determine the normal feature vector of the current frame feature vector; wherein, if the encoding modal of the encoder is a time sequence modal, the normal feature vector of the current frame is calculated by using a sliding window method, and if it is a non-time sequence modal, the feature mean of the same type of actual data in the application scene is used as the normal feature vector of the current frame.

[0068] In the embodiment of the application, the time sequence modal refers to a modal with time continuity (such as audio, video frame sequence, radar point cloud time sequence data, and sensor dynamic signal), and the normal feature vector thereof is generated based on the historical feature smoothing value within the sliding window. The size of the sliding window is set to 5 frames (which can be dynamically adjusted according to the modal sampling rate, such as setting the window of high-frequency sensor data to 10 frames), and the window contains the historical features of the previous 4 frames of the current frame (to avoid containing the current frame itself to prevent pollution of the benchmark).

[0069] In the embodiment of the application, the normal feature vector F t-normal = (1 / k) Σ t-1 r=t-k F r , F r is the historical feature vector of the rth frame, r takes a value of t-k to t-1, k is the window size, and t is the current frame number. Wherein, when t≤k (such as the first 5 frames), the feature mean of the previous t-1 frames is fused with the initial feature mean of the same sample to generate the benchmark, so as to avoid the empty window problem when t=1 without historical data.

[0070] The non-time sequence modal refers to a static modal without time correlation (such as text, single image, and single frame depth map), and the normal feature vector thereof is generated based on the feature distribution of the same type of sample in the same scene: Pre-construct a scene feature library: classified according to application scenes (such as traffic monitoring and medical diagnosis), collect high-quality samples of non-time sequence modals under each scene (verified by artificial without distortion), calculate the feature mean and variance of the same type of sample, and form a scene benchmark library; The normal feature vector directly calls the feature mean of the same type of sample under the current scene: F normal =Avg scene，type , Avg scene，type is the feature mean of the modal type (such as “accident scene image”) under the current scene (such as “traffic accident”).

[0071] S220, calculate the difference value between the current frame feature vector and the normal feature vector at the output end of the current encoder, and if the difference value is greater than a preset threshold, determine that the current frame feature vector has feature distortion.

[0072] In the embodiment of the application, the difference value D between the current frame feature vector and the normal feature vector satisfies the following condition: D= (1 / d) Σd s=1 |F normal，s -F current，s |, where F normal，s F is the s-th element in the normal feature vector. current，s This is the s-th element of the feature vector output by the current encoder, where s takes values ​​from 1 to d, and d is the dimension of the feature vector, for example, 1024 dimensions.

[0073] In one embodiment of the present invention, the preset threshold can be a fixed value, such as 0.15.

[0074] In another embodiment of the present invention, the preset threshold can be dynamically adjusted according to the modality type and feature level (different modalities / different coding layers have different feature stability): Temporal modalities (such as audio and video): The threshold for the bottom coding layer (first 3 layers) is set to 0.2 (features contain more details, allowing for greater fluctuations), and the threshold for the top coding layer (last 3 layers) is set to 0.15 (features are more abstract, requiring strict control of fluctuations). Non-temporal modalities (such as text and images): The threshold for the bottom coding layer is set to 0.18, and the threshold for the top coding layer is set to 0.12 (static features have higher stability, so the threshold is more stringent).

[0075] S230, if feature distortion is detected, an adaptive error correction mechanism is immediately triggered. If the difference value of the feature vector of the current frame is still greater than the preset threshold after triggering the error correction mechanism for m consecutive times, for example 3 times, the multimodal data to be processed corresponding to the encoder is marked as abnormal data and temporarily stored in the abnormal database. The subsequent processing of the abnormal data is suspended, and the processing is restarted after the data quality is manually reviewed and confirmed.

[0076] In this embodiment of the invention, different error correction mechanisms are adopted according to the degree of distortion to avoid efficiency loss caused by single recoding. Specifically, S230 includes: S2301, if D0<D≤h×D0, execute S2302, if D>h×D0, execute S2303; where D0 is a preset threshold and h is a preset value, for example h=1.5.

[0077] S2302, an error correction mechanism that triggers intra-layer residual correction instead of direct recoding, includes: calculating the residual Δ between the current layer features and the previous layer's output features; and correcting the current features using the residual compensation: F corrected =F current -0.5△ is used to weaken abnormal fluctuations; D is recalculated after correction. If D≤D0, the subsequent processing continues; if D>D0, S2302 is executed.

[0078] S2303, triggers a re-encoding mechanism, including: The first re-encoding: enable the standby encoder to re-encode the original data, wherein the text mode is replaced by RoBERTa, the image mode is replaced by Swin-Transformer, the audio mode is replaced by YAMNet, etc., to generate new features F1; calculate the difference value D1 of F1, if D1≤D0, continue the candidate processing; otherwise, enter the next re-encoding.

[0079] The second re-encoding: adjust the encoding parameters to optimize feature extraction, wherein the image compression rate is reduced to 80% of the original resolution, the audio sampling rate is increased to 44.1kHz, and the text retains more context windows (such as from 512 tokens to 1024 tokens); generate new features F2 based on the new parameters, calculate the difference value D2 of F2, if D2≤D0, continue the candidate processing; otherwise, enter the next re-encoding.

[0080] The gth re-encoding (2

[0081] S2304, the following operations are performed: Automatically mark the data unit as "abnormal", record key information, including distortion type (such as sensor noise, transmission packet loss, data damage), D value of each time, abnormal feature dimension index (such as audio high frequency band, image edge area); Temporary storage in the exception database, suspend the subsequent processing flow of the data unit (avoid distortion feature pollution in cross-modal fusion link), at the same time trigger multi-level alarm (system message real-time notification + daily exception summary email); Manual review and scene processing: if it is determined to be "repairable" (such as light noise pollution), call special tools (Webrtcvad for audio noise reduction, LaMa model for image repair) for processing, and restart the whole process from S100; if it is determined to be "unrepairable" (such as data frame loss, serious damage), mark it as "invalid data" and archive, and associate the log for subsequent modal acquisition equipment optimization reference.

[0082] S300, based on the fusion features of each modal data, realize cross-modal information interaction, and output cross-modal feature vectors of each modal.

[0083] The step is used to realize cross-modal information interaction based on the fusion features of each modal data in the interaction layer of the parallel Transformer encoder, and output cross-modal feature vectors of each modal.

[0084] The input of the interaction layer is the fusion features of each modal data output by the encoding layer S200, which have the following commonalities and characteristics: Commonalities: all contain deep fusion results of basic features + semantic anchor features (such as text fusion features containing “original text semantics + entity / event anchor”, image fusion features containing “pixel features + target / region anchor”), and the dimension is unified to d feat =1024 (convenient for cross-modal calculation); Characteristics: the fusion features of the time sequence modal (text, audio, video) contain time sequence position encoding (such as the sequence position of text tokens and the timestamp of audio frames), and the feature dimension is [T, d feat ], where T is the time sequence length; the fusion features of the spatial modal (image, point cloud, depth map) contain spatial position encoding (such as pixel coordinates and point cloud three-dimensional coordinates), and the feature dimension is [H×W, d feat ], H×W is the spatial flattening dimension).

[0085] Further, S300 specifically includes: S301, for any modal data X, taking the fusion feature thereof as a query vector, and performing one-way cross-attention calculation with the semantic anchor features of other modal data in the to-be-processed multi-modal data respectively, to generate one-way cross-attention results of the modal data X for each other modal data.

[0086] The one-way cross-attention results of the modal data X for each other modal data can be obtained through the aforementioned cross-modal attention mechanism.

[0087] S302, weighting and fusing the fusion feature of the modal data X and all one-way cross-attention results generated in S301 to generate cross-modal intermediate features.

[0088] The cross-modal intermediate features Mid X of the modal data X satisfy the following conditions: Mid X =α•F x +∑ u b=1 β h •Output Xh .

[0089] Wherein, α+∑ u b=1 β h =1. α is the weight of the fusion feature F X of the modal data X, α≥0.4, β his learnable parameter, which is optimized by model training, h is an integer from 1 to u, u is the number of other modalities, Output Xh is the one-way cross-attention result of X on the hth other modality data.

[0090] S303, intra-modal self-attention optimization is performed on the intermediate feature of modality X to resolve cross-modal information conflict, and a cross-modal feature vector of modality X is output.

[0091] The intermediate feature may have cross-modal information conflict (such as the text obtaining "accident time 8:00" from the image and "8:05" from the audio), which needs to be resolved by the intra-modal self-attention layer for conflict resolution and consistency enhancement, including: (1) taking the intermediate feature Mid X of modality X as Q, K and V, the self-attention optimization feature of modality data X is calculated through the self-attention mechanism. The self-attention mechanism learns Mid X the correlation weight of each feature unit inside, and the model focuses on more reliable information such as the "timestamp anchor" 8:00 of the image, and corrects the conflict with the audio.

[0092] (2) performing layer normalization on the self-attention optimization feature of modality data X, and connecting it with Mid X after residual connection, the cross-modal feature vector of modality data X is obtained.

[0093] The final output of each modality cross-modal feature vector has the following core characteristics: Preserve its own modality characteristics: through the weight design of a≥0.4, ensure that the core information (such as the semantic logic of the text and the visual details of the image) of the encoding layer fusion feature is included; Integrate cross-modal correlation: through one-way cross-attention calculation and weighted fusion, the semantic anchor correlation information (such as the "event-visual" correlation between text and image) with all other modalities is integrated; Strong semantic consistency: through self-attention optimization, cross-modal information conflict is effectively resolved, and the internal logic of the feature is self-consistent, providing high-quality input for subsequent cross-modal fusion tasks.

[0094] S400, obtain the relationship type between each modality data, and determine the corresponding modality fusion weight based on the obtained relationship type, perform differential fusion on the cross-modal feature vector of each modality data, and obtain the current global fusion feature.

[0095] Further, S400 specifically includes: S410, obtaining a sharing degree score S(M1, M2) between any two modalities, if S(M1, M2) ≥ S0, it indicates that the semantics and structure of the two modalities are closely related, such as text and image describing the same event, and the two modalities are determined as sharing relationship, otherwise, it indicates that the correlation is weak, such as background audio and foreground text, and it is determined as independent relationship.

[0096] wherein, S(M1, M2) = a x Sim(V1, V2) + (1-a) x C(M1, M2), wherein M1 is one of the two modalities, M2 is the other of the two modalities, V1 is the cross-modal feature vector of M1, V2 is the cross-modal feature vector of M2, Sim(V1, V2) is the semantic vector cosine similarity of M1 and M2, C(M1, M2) is the modal temporal consistency or logical structure consistency score between M1 and M2, if M1 and M2 are temporal modalities (such as audio and video), C(M1, M2) is the timestamp overlap degree (value range 0-1, the higher the overlap, the greater the score); if M1 and M2 are spatial or mixed modalities (such as text and image), C(M1, M2) is the logical structure matching degree (such as the hierarchical correspondence degree of text entities and image targets, value range 0-1); S0 is a preset sharing degree threshold, for example, S0 = 0.6, a is a weight coefficient, 0 < a < 1, which is balanced by training and optimization to balance the influence of semantic and structural association.

[0097] S420, for the two modalities determined as sharing relationship, the weight of M1 absorbing M2 information and the weight of M2 absorbing M1 information are determined by the following steps, which are used to generate the global association features of the two modalities respectively. S4201, based on the semantic attributes (such as event time, place, subject) of the core association entity (such as "vehicle chasing event") pointed to by the two modalities, the directly related feature subsets (such as "chasing car" entity features in text and "vehicle motion trajectory" features in image) are selected from V1 and V2, and the redundant noise features (such as irrelevant auxiliary words in text and background sky features in image) are removed, to obtain the filtered features VF1 and VF2 (the dimension is consistent with the original feature, and the noise position is set to 0).

[0098] S4202, the cross-attention mechanism is used to calculate the association degree of the feature pairs in VF1 and VF2, which specifically includes: (1) taking VF1 as Query (Q), VF2 as Key (K) and Value (V), to calculate the first group of association weights Attn1, Attn1[i1][i2] represents the attention degree of the i1th sub-feature of VF1 to the i2th sub-feature of VF2, wherein i1 is the sub-feature index of VF1, i2 is the sub-feature index of VF2, and the value range is 1 to the feature dimension n. (2) Take VF2 as Query (Q) and VF1 as Key (K) and Value (V) to calculate the second group of attention weights Attn2, Attn2[i2][i1] representing the attention degree of the i2th sub-feature of VF2 to the i1th sub-feature of VF1; (3) The correlation degree R(i1, i2) of the feature pair formed by the i1th sub-feature in VF1 and the i2th sub-feature in VF2 = (Attn1[i1][i2] + Attn2[i2][i1]) / 2.

[0099] S4203, based on R(i1, i2), give the initial weight of absorbing the information of the other mode to the two modes: The initial weight W1[i] of M1 absorbing M2 information is positively correlated with the maximum correlation degree of VF1[i1] in VF2, that is, W1[i] = max i2 (R(i1, i2)); The initial weight W2[i2] of M2 absorbing M1 information is positively correlated with the maximum correlation degree of VF2[i2] in VF1, that is, W2[i2] = max i1 (R(i1, i2)); Normalize W1 and W2 respectively (the sum of weights is 1) to obtain the initial fusion weight vectors W1 init and W2 init .

[0100] S4204, adjust the initial weight reversely to ensure that the complementary features of the two modes (such as features in VF1 but not in VF2) are not weakened, with the goal of minimizing the modal complementary loss function.

[0101] Wherein, the modal complementary loss function satisfies the following conditions: L comp =∑ n i=1 |f1(i)-f2(i)|•I(f1(i)≠0∨f2(i)≠0), wherein f1(i) is the i-th element of the feature vector of one mode in the shared relationship after S4201 processing, f2(i) is the i-th element of the feature vector of the other mode in the shared relationship after S4201 processing, n is the dimension of the feature vector, I(•) is an indicator function, ∨ is a logical or operator, I(•) takes the value 1 when f1(i) or f2(i) is not zero, otherwise 0.

[0102] The optimization process is: minimize L comp by gradient descent to adjust W1 init and W2 init to obtain the optimized fusion weights W1 opt and W2 opt .

[0103] S4205, update the features of M1 and M2 based on W1 opt and W2 opt , respectively, wherein the associated feature of M1 is updated as: V1 update = V1 + W1 opt ⊙V2; and the associated feature of M2 is updated as: V2 update = V2 + W2 opt ⊙V1.

[0104] S430, for the two modalities M1 and M2 determined as independent relationship, a weak fusion strategy is adopted to avoid irrelevant information interference, and the associated features of the two modalities are updated: V1 and V2 remain independent coding state, without cross-modal feature deep interaction (such as cross-attention calculation); The weight of M1 absorbing M2 information and the weight of M2 absorbing M1 information are both decayed according to the following formula: W weak =W0×(1-S(M1, M2)) calculation, wherein W weak is the modal fusion weight, and W0 is the basic weight, which can be 0.3; In the weight vector, the elements related to the cross-association of the two independent modalities (such as the weights corresponding to the sub-features irrelevant to M2 in M1) are set to 0 to prevent irrelevant feature pollution; Based on W weak , the features of M1 and M2 are updated, respectively, wherein the associated feature of M1 is updated as: V1 update = V1 + W weak ⊙V2; and the associated feature of M2 is updated as: V2 update = V2 + W weak ⊙V1. S440, for each modality Mx, the global associated feature of Mx is generated by interacting with and accumulating updates of all other modalities My (y≠x) in the multi-modal data to be processed, and the specific steps are: S4401, taking the cross-modal feature vector Vx of Mx as the initial feature, denoted as Vx current = Vx; S4402, for each other modality My, traverse in a predetermined order, such as My1, My2, …, Myu: If Mx and My are shared relationship, call Vx update output by S4205, i.e. Vx update = Vx current + Wx opt ⊙Vy, Wx opt is the optimized weight of Mx absorbing My, and Vy is the cross-modal feature vector of My; If Mx and My are independent relationship, call Vxupdate , i.e. Vx update = Vx current +Wx weak ⊙ Vy, Wx weak is the weak fusion weight of Mx absorbing My; Update the current feature: Vx current = Vx update (integrate the accumulated information of My into the feature of Mx); S4403, determine the global correlation feature: when Mx interacts with all My, Vx current , i.e. the global correlation feature of Mx contains the correlation information with all modalities, and the dimension is consistent with Vx.

[0105] If Mx and other modalities My (y≠x) are in a shared relationship, use Vx update output by S4205 as the correlation update result of Mx and My, if it is an independent relationship, use Vx update output by S430 as the correlation update result of Mx and My; the correlation update result of Mx and all other modalities is used as the global correlation feature of Mx.

[0106] S450, concatenate the global correlation features of all modalities in a preset order (such as time sequence modality→spatial modality) to obtain a concatenated feature with a dimension of [T total , d feat ], T total is the total length of all modality features, and d feat =1024; and uniformly distribute the concatenated feature through a linear transformation layer (such as a full connection layer or a 1×1 convolution) to finally output the current global fusion feature.

[0107] S500, detect the semantic consistency of the global fusion feature; if there is a semantic conflict, dynamically adjust the current modality fusion weight and return to S400, otherwise execute S600.

[0108] In the embodiment of the application, the global fusion feature is input into a lightweight large language model LLM (such as distilled Vicuna-7B) to guide the model to detect the semantic conflict between modalities through prompt, which specifically includes: Input definition: global fusion feature F global is converted into a natural language description sequence (such as “text: ‘vehicle 8:00 collision’; image: ‘collision area at intersection’; audio: ‘8:05 brake sound’”); Prompt design: The fixed template is "Please judge whether the following multi-modal information has semantic conflict (such as inconsistency of time, place, event subject), output conflict score C (t), range is -1 to 1 (-1 represents complete conflict, 1 represents complete consistency, 0 represents no obvious conflict): {input multi-modal description sequence}"; Output result: LLM outputs the semantic conflict score C (t) of the current time t (t is the iteration number, and the initial t = 1), which is used to determine whether there is conflict: If C (t) < 0: it is determined that there is a semantic conflict (the lower the score, the more serious the conflict); If C (t) >= 0: it is determined that the semantics are consistent and do not need to be adjusted.

[0109] In S500, if it is detected that the global fusion feature has semantic conflict, the modal fusion weight is adjusted in the following manner: Wt d =Wt x (1 / (1+e -λ×C（t） ), Wt is the modal fusion weight of the current time t, including the shared relationship optimization weight output by S4204 and the independent relationship weak fusion weight output by S430, Wt d is the adjusted modal fusion weight of the current time t, e is a natural constant, and lambda is an adjustment coefficient, which can be 5. Through experimental verification: when C (t) =-1 (complete conflict), (1 / (1+e -λ×C（t） ) approximately equal to 0.0067, and the weight is reduced to 0.67% of the original weight; when C (t) =-0.5 (moderate conflict), (1 / (1+e -λ×C（t） ) approximately equal to 0.0759, and the weight is reduced to 7.59% of the original weight, so that the more serious the conflict, the more significant the weight attenuation.

[0110] In the embodiment of the application, the weight of the conflict mode is dynamically compressed (the more serious the conflict, the greater the weight compression ratio) through (1 / (1+e -λ×C（t） ), so as to reduce the influence of the conflict mode on the fusion result and preferentially retain the modal information with consistent semantics. In the embodiment of the application, the fusion process of S410 to S450 is re-executed using the adjusted weight, a new global fusion feature is generated, and the detection step of the semantic conflict is repeated until one of the following termination conditions is met: Condition 1: the newly detected conflict score C (t+1) >= 0 (semantic consistency), and the current global fusion feature is output as the corrected global fusion feature; Condition 2: the iteration number reaches a preset maximum threshold, and the global fusion feature of the last iteration is output. The preset maximum threshold can be 3 times, and through experimental verification: 3 iterations can cover more than 95% of conflict resolution scenarios, avoiding efficiency loss caused by excessive iteration.

[0111] S600, converting the global fusion features without semantic conflicts into semantic information, and supplementing fine-grained data according to the fine-grained requirements of the extraction instruction to form a preliminary semantic extraction result.

[0112] S600 specifically includes: S601, converting the global fusion features without semantic conflicts into lightweight LLM (such as distilled Vicuna-7B) resolvable semantic signals through a feature-language mapping layer. The core operation is as follows: Input preprocessing: modal dimension separation and semantic label binding are performed on the global fusion features. Specifically, the global correlation feature subsets (such as text subset, image subset, and audio subset) of each modality are split according to the splicing order (temporal modality→spatial modality) of S450, and a modality type label (such as [TEXT], [IMG], and [AUDIO]) is attached to each subset to ensure that the LLM can identify the feature source. Mapping layer structure: a “bilinear projection + layer normalization” design is adopted to uniformly map each modality feature subset to the input dimension of the LLM (such as the input dimension d_model=4096 of Vicuna-7B): SemSignalx=LayerNorm(Wx•Fx+bx).

[0113] Where x represents the modality type, Wx is the modality-specific projection matrix (dimension [4096, 1024]), bx is the bias term, and SemSignalx is the single-modality semantic signal.

[0114] Global semantic signal integration: the SemSignalx of each modality is spliced according to “temporal logic + spatial logic” (such as text semantic signal→image spatial semantic signal→audio temporal semantic signal) to form the final global semantic signal SemSignal global (dimension [T_sem, 4096], T_sem is the semantic sequence length), ensuring that the signal contains cross-modality temporal / spatial correlations.

[0115] S602, inputting the global semantic signal SemSignal global into the lightweight LLM to guide the model to extract abstract semantics from the global fusion features through a preset parsing template, specifically including: Template design: The prompt is fixed as "Based on the following multi-modal semantic signals, extract 3 types of core information: 1. Entities (including types and attributes, such as [Entity: Vehicle, Type: Transportation, Attribute: Color = Red]); 2. Relationships (including subject-predicate-object triples, such as [Vehicle - Collision - Guardrail]); 3. Events (including time, location, and subject, such as [Event: Vehicle Collision, Time: 8:00, Location: Intersection, Subject: Red Sedan]): {SemSignal_global}"; LLM output format: The model outputs the parsing result in a structured JSON format.

[0116] Preliminary semantic extraction result definition: The structured JSON result output by the LLM is denoted as Init_Sem_Result, which serves as the basis for subsequent fine-grained supplementation.

[0117] S603, through the parallel embedding interface of the sub-module, according to the fine-grained requirements of the extraction instruction (such as "extract the pixel coordinates of the collision area" "extract the timestamp of the brake sound"), call the preset special sub-module, synchronize the fine-grained data to Init_Sem_Result, the specific process is as follows: Fine-grained requirement judgment: Analyze the "fine-grained keywords" in the user's extraction instruction (such as "pixel coordinates" corresponding to visual fine-grained requirements, "timestamp" corresponding to audio fine-grained requirements), and match the corresponding special sub-module (sub-module library presets SEEM visual segmentation module, SHAS audio segmentation module, etc.); Sub-module calling rules: If the requirement is visual fine-grained (such as "collision area pixel coordinates"): Call the SEEM module, input the original image data of S200 encoding layer (including spatial position encoding), output the pixel coordinate range of the target area (such as [x1:200, y1:300, x2:400, y2:500]); If the requirement is audio fine-grained (such as "brake sound timestamp"): Call the SHAS module, input the original audio data of S200 encoding layer (including time sequence position encoding), output the timestamp of the target audio segment (such as [start:1.2s, end:1.5s]); Sub-module and parallel encoding process synchronization: The input of the sub-module is directly associated with the original modal data of S200 (avoiding repeated encoding), and the output result is real-time integrated into SemSignal_global through "modal index alignment" (such as the pixel coordinates output by SEEM are associated with the spatial index of the global feature of the image); Fine-grained data integration: The fine-grained data output by the sub-module is supplemented to the corresponding field of Init_Sem_Result in the form of "attribute expansion".

[0118] S604: Format checking is performed on Init_Sem_Result (ensure field integrity, data type consistency), and finally the structured preliminary semantic extraction result (supporting common formats such as JSON, XML, etc.) is output, providing a basis for subsequent semantic refinement or task application (such as multi-modal question answering, event tracing).

[0119] S700, based on the knowledge graph, the attribute integrity of the preliminary semantic extraction result is checked, and if the attribute is missing, the completion process is triggered, and a check result with complete integrity is obtained.

[0120] Further, S700 specifically includes: S710, each entity in the preliminary semantic extraction result is matched with a standard entity type in the knowledge graph, and a corresponding matching result is obtained.

[0121] In the embodiment of the application, the knowledge graph is defined as a preset domain knowledge graph (such as a “traffic accident multi-modal knowledge graph”), the structure of which includes a “entity type-standard attribute” hierarchical relationship (for example, as shown in Table 1 below), and each entity type is bound to a unique standard attribute set (the attribute type includes “mandatory attribute” and “optional attribute”, and only the mandatory attribute is counted in the integrity calculation).

[0122] Table 1 Entity Type Standard Mandatory Attributes in Knowledge Graph Total Number of Standard Attributes (Nt) Vehicle (Transportation) Model, Color, License Plate, Age 4 Collision Event (Event) Occurrence Time, Occurrence Location, Parties Involved, Event Result 4 Road Guardrail (Facility) Material, Location Coordinates, Belonging Road Section 3 In the embodiment of the application, the matching result can include a matching successful standard entity type or a “to-be-confirmed entity” label for a matching failure. Specifically, for each entity (such as “red sedan” and “intersection collision event”) in the preliminary semantic extraction result Init_Sem_Result output by S604, the entity name and type label (such as [entity: vehicle, type: traffic tool]) are matched with the corresponding standard entity type (such as “red sedan” matching “vehicle (traffic tool) type”) in the knowledge graph. If the entity (such as “unknown object”) in the preliminary result cannot be matched to a standard type in the knowledge graph, it is marked as “to-be-confirmed entity” and is not included in the attribute integrity calculation (subsequently supplemented by manual checking or a general sub-module).

[0123] S720, based on the aligned standard entity type, the attribute integrity representation value P of each entity in the preliminary semantic result is calculated.

[0124] Wherein, P=Ne / Nt×100%, Ne is the number of standard mandatory attributes successfully extracted in the preliminary semantic extraction result (which needs to satisfy the attribute value is not empty and the format is valid, such as “color=red” is valid extraction, “color=” is invalid extraction, and is not included in Ne), and Nt is the total number of standard mandatory attributes of the entity in the knowledge graph.

[0125] If the preliminary semantic extraction result contains multiple types of entities, the "overall attribute integrity representation value" takes the minimum value among the P values of all types of entities, ensuring that the optional attributes of each type of core entity meet the basic integrity requirement (e.g., "vehicle" P=60%, "collision event" P=100%, then overall P=60%).

[0126] S730, if P < P0, call the preset fine-grained submodule to extract the missing attributes from the multi-modal data to be processed, and recompute P after completion. If P ≥ P0, directly take the preliminary semantic extraction result as the verification result of the integrity verification.

[0127] Wherein, P0 is a preset integrity representation threshold, for example, 80%, through domain experiment verification: the threshold can balance the attribute integrity requirement and extraction efficiency, avoid resource waste caused by excessive completion.

[0128] Specifically, if the overall attribute integrity representation value P ≥ P0: directly take the preliminary semantic extraction result Init_Sem_Result as the verification result of the integrity verification; If the overall attribute integrity representation value P < P0: (1) Missing attribute identification: for each type of entity with P < P0, identify its missing standard optional attributes (e.g., "vehicle" entity missing "license plate" and "service life" attributes); (2) Submodule mapping and calling: establish a directional mapping relationship between "missing attributes-fine-grained submodules" (for example: "vehicle" missing "license plate" calls SEEM visual segmentation + OCR recognition submodule, "collision event" missing "occurrence time" calls SHAS audio segmentation + timestamp alignment submodule), the input of the submodule comes from the original modal data (such as original image, audio) of the S200 encoding layer or the global correlation features (including spatial / time encoding, avoiding repeated calculation) output by S450; (3) Attribute completion and update: call the corresponding submodule to extract the missing attributes, supplement them to the preliminary semantic extraction result in the form of "attribute expansion", and update to Sem_Result_updated; (4) Iterative verification: based on Sem_Result_updated, repeat the calculation steps of S720 to get a new overall attribute integrity representation value Pnew; repeat the process of "missing identification-submodule calling-completion updating-Pnew calculation" until one of the following termination conditions is met: Condition 1: Pnew ≥ P0; Condition 2: the number of completion iterations reaches a preset maximum threshold (domain experiment verification: 2 iterations can cover more than 90% of attribute completion requirements, attributes that exceed 2 iterations and still cannot meet the requirements are mostly information that cannot be extracted from the data, and do not need to continue iteration). The result output: after meeting the termination condition, output the check result of attribute integrity (Sem_Result_updated or the result of the last iteration).

[0129] S800, based on the check result, generate a multi-granularity unified information extraction result containing global extraction result and fine-granularity grounding information.

[0130] In the embodiment of the application, the multi-granularity unified information extraction result is composed of two parts: global extraction result and fine-granularity grounding information, wherein: the global extraction result is the structured abstract semantic information in the check result, including entity list (containing entity name, type and core attribute), relationship triple (subject-predicate-object association), event graph (event elements and associated relationships), which is used to meet the needs of fast application scenarios (such as multi-modal summary, event overview); the fine-granularity grounding information is the original modal fine-granularity data (such as pixel-level segmentation mask of entity in image, timestamp boundary of event fragment in audio, character-level positioning coordinates of entity in text) bound to the entity / event in the global extraction result, which is used to support high-precision application scenarios (such as entity tracing, detail verification).

[0131] The association between fine-granularity grounding information and global extraction result is established through the grounding mapping layer: for each entity or event in the global extraction result, at least one corresponding original modal fine-granularity data (such as image segmentation mask for "vehicle" entity, audio timestamp for "collision event") is bound, and the accuracy of the association is ensured through optimization of mapping precision index.

[0132] The mapping precision is optimized by the following formula: Lg=∑ m v=1 IoU(Mv, Bv).

[0133] Wherein, m is the total number of the association pairs of the fine-grained grounding information and the global result, that is, the total number of entity / event-fine-grained data pairs that need to establish a mapping relationship; v is the index of the association pair, the value range is 1 to m, used to traverse each association pair; Mv is the original modal fine-grained data in the vth association pair (such as the segmentation mask of the entity in the image, represented by a set of pixel coordinates; the time mask of the event segment in the audio, represented by [start, end] timestamp), Bv is the reference boundary of the entity / event corresponding to the global extraction result in the vth association pair (such as the character position boundary of the entity in the text, represented by [start_idx, end_idx]; the time boundary of the event description in the audio-to-text, represented by [text_start, text_end] timestamp), IoU() is the intersection over union of Mv and Bv, which is the pixel overlap degree in the image modal, and the time overlap degree in the time sequence modal, the value range is 0 to 1, the larger the value, the higher the matching degree of the fine-grained information and the global result.

[0134] By maximizing Lg, that is, maximizing the sum of the intersection over union of all association pairs, the mapping accuracy of the fine-grained grounding information and the global extraction result is improved, and finally the multi-granularity unified information extraction result with accurate association and complete hierarchy is output.

[0135] The embodiment of the application also provides an electronic device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the application.

[0136] The embodiment of the application also provides a computer readable storage medium storing computer executable instructions, and the computer executable instructions are used to execute the method described in the embodiment of the application.

[0137] It should be understood that the steps can be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the application can be executed in parallel, in sequence or in different order, as long as the desired results of the technical solutions disclosed in the application can be achieved, which is not limited herein.

[0138] The above specific embodiments do not constitute a limitation on the protection scope of the application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A method for dynamic adaptation and collaborative extraction of large models based on a unified framework, characterized in that, The method includes the following steps: S100 performs dynamic modality recognition and adaptation on the multimodal data to be processed, and performs feature encoding on each modality data to obtain the basic features of each modality data; S200 identifies the semantic anchor points of each modal data, concatenates the basic features and semantic anchor points of each modal data, and inputs them into the corresponding parallel Transformer encoder for encoding to obtain the fused features of each modal data; at the same time, a monitoring node is set at the output of each layer of each encoder to monitor and correct feature distortion in real time. S300, based on the fusion features of data from each modality, realizes cross-modal information interaction and outputs cross-modal feature vectors of each modality; S400: Obtain the relationship type between each modal data, then determine the corresponding modal fusion weight based on the obtained relationship type, and perform differential fusion on the cross-modal feature vectors of each modal data to obtain the current global fusion feature; S500: Detect the semantic consistency of the global fusion features; if semantic conflicts exist, dynamically adjust the current fusion weights of each modality and return to S400; otherwise, execute S600. S600 converts globally fused features without semantic conflicts into semantic information, and supplements fine-grained data according to the fine-grained requirements of the extraction instructions to form preliminary semantic extraction results; S700 verifies the completeness of attributes in the preliminary semantic extraction results based on the knowledge graph. If an attribute is missing, a completion process is triggered to obtain a verification result that meets the completeness standard. S800, based on the verification result, generates a multi-granularity unified information extraction result that includes global extraction results and fine-grained grounding information.

2. The method according to claim 1, characterized in that, S100 specifically includes: S110, based on modal feature entropy values ​​and a preset multimodal embedding model, automatically identify the modal type of the multimodal data to be processed, wherein the modal type includes known modalities and unknown modalities; S120, feature encoding and unified embedding processing are performed on the identified known modalities and unknown modalities respectively to obtain the initial features of the known modalities and unknown modalities respectively; among them, for unknown modalities, the corresponding feature extraction submodule is triggered according to the physical properties of the unknown modalities, and after performing preliminary feature extraction, it is mapped to the unified embedding space of the preset multimodal embedding model to form the initial features of the unknown modalities. S130, dynamically adjust the initial dimensions of the known and unknown modalities to obtain a standard tensor in a preset format, which is the basic feature of each modal data.

3. The method according to claim 2, characterized in that, in, In S120, if the unknown mode is a continuous signal, the feature extraction submodule automatically uses Fourier transform and wavelet analysis tools to extract frequency domain and time domain features; if the unknown mode is a discrete structure, the feature extraction submodule captures local structural features through character-level embedding and n-gram statistical methods.

4. The method according to claim 1, characterized in that, In the S200, a monitoring node is set at the output of each layer of each encoder to monitor and correct feature distortions in real time, specifically including: For each encoder, the normal feature vector of the current frame feature vector is determined according to its corresponding coding mode type. If the coding mode of the encoder is a temporal mode, the normal feature vector of the current frame is calculated using the sliding window method. If it is a non-temporal mode, the feature mean of the same type of actual data in the application scenario is used as the normal feature vector of the current frame. Calculate the difference between the current frame feature vector and the normal feature vector at the current encoder output. If the difference is greater than a preset threshold, it is determined that the current frame feature vector has feature distortion. If feature distortion is detected, an adaptive error correction mechanism is immediately triggered. If the difference value of the feature vector of the current frame is still greater than the preset threshold after the error correction mechanism is triggered m times consecutively, the multimodal data to be processed corresponding to the encoder is marked as abnormal data and temporarily stored in the abnormal database. The subsequent processing of the abnormal data is suspended, and a decision is made on whether to restart the processing after the data quality is manually reviewed and confirmed.

5. The method according to claim 1, characterized in that, The S400 specifically includes: S410, obtain the sharing score S(M1, M2) between any two modalities. If S(M1, M2) ≥ S0, determine that the two modalities are in a sharing relationship; otherwise, determine that they are in an independent relationship. Wherein, S0 is a preset sharing threshold; M1 is one of the two modalities, and M2 is the other of the two modalities. S420, For two modalities determined to have a sharing relationship, the following steps are used to determine the weights of M1 absorbing information from M2 and the weights of M2 absorbing information from M1, which are then used to generate the global association features for each of the two modalities: S4201, based on the semantic attributes of the core related entities that both modalities point to, select directly related feature subsets from the cross-modal feature vector V1 of M1 and the cross-modal feature vector V2 of M2, remove redundant noise features, and obtain the selected features VF1 and VF2. S4202 uses a cross-attention mechanism to calculate the correlation between feature pairs in VF1 and VF2; S4203, based on the correlation between feature pairs in VF1 and VF2, assigns initial weights W1 to the two modalities to absorb information from each other. init and W2 init ; S4204, with the objective of minimizing the modal complementarity loss function, adjusts the initial weights in reverse to obtain the optimized fusion weights W1. opt and W2 opt ; S4205, based on W1 opt and W2 opt Update the features of M1 and M2 respectively; S430, for two modalities M1 and M2 that are determined to be independent, a weak fusion strategy is adopted to avoid interference from irrelevant information and update the correlation features of the two modalities: V1 and V2 remain in independent encoding states and do not engage in cross-modal feature depth interaction; Both the weight of M1 absorbing information from M2 and the weight of M2 absorbing information from M1 decrease according to the following formula: W weak =W0×(1-S(M1, M2)) is calculated, where W weak W0 is the base weight for modal fusion; In the weight vector, the elements that are cross-correlated with the two independent modes are set to 0 to prevent contamination by irrelevant features; Based on W weak Update the features of M1 and M2 respectively; S440, for each modality Mx, generate global association features of Mx by interacting with and cumulatively updating all other modalities My (y≠x) in the multimodal data to be processed; S450 concatenates the global associated features of all modalities in a preset order to obtain concatenated features, and then unifies the feature distribution of the concatenated features through a linear transformation layer, finally outputting the current global fusion features.

6. The method according to claim 5, characterized in that, The modal complementarity loss function satisfies the following condition: L comp =∑ n i=1 |f1(i)-f2(i)|•I(f1(i)≠0∨f2(i)≠0), where f1(i) is the i-th element of the feature vector of one modality in the shared relation after processing by S4201, f2(i) is the i-th element of the feature vector of another modality in the shared relation after processing by S4201, n is the dimension of the feature vector, I() is the indicator function, ∨ is the logical OR operator, where if f1(i)≠0 and / or f2(i)≠0, I()=1, if f1(i)=0 and f2(i)=0, I()=0, and • represents dot product.

7. The method according to claim 1, characterized in that, In S500, if a semantic conflict is detected in the global fusion features, the fusion weights of each modality are adjusted as follows: Wt d =Wt×(1 / (1+e -λ×C（t） Wt represents the modal fusion weights at time t. d Let e ​​be the adjusted modal fusion weight at time t, λ be the natural constant, λ be the adjustment coefficient, and C(t) be the semantic conflict score at time t.

8. The method according to claim 1, characterized in that, The S700 specifically includes: S710, Match each entity in the preliminary semantic extraction results with the standard entity types in the knowledge graph to obtain the corresponding matching results; S720, based on the aligned standard entity type, calculate the attribute integrity representation value P for each entity in the preliminary semantic results; S730, if P < P0, call the preset fine-grained submodule to extract missing attributes from the multimodal data to be processed, complete them and recalculate P until P ≥ P0; if P ≥ P0, directly use the preliminary semantic extraction result as the verification result of the integrity verification, where P0 is the preset integrity characterization value threshold.

Images