Multi-level fine-grained semantic alignment method based on cross-modal information fusion

Abstract

The invention discloses a multi-level fine-grained semantic alignment method based on cross-modal information fusion, and belongs to the technical field of artificial intelligence. The method comprises the following steps: a multi-modal data preprocessing module preprocesses acquired multi-modal input to generate a multi-modal intermediate expression format; inputting the multi-modal intermediate expression format into an encoder corresponding to each modal, and mapping an embedded expression vector corresponding to each modal to a shared semantic space by a unified projection module to form a feature expression corresponding to each modal; segmenting the feature expression corresponding to each mode into minimum semantic units, determining semantic feature granularity according to a to-be-executed task, and aggregating the minimum semantic units in combination with semantic similarity, a space-time proximity relationship and a cross-mode mapping rule to form semantic blocks; and performing feature alignment and feature fusion on the feature expressions corresponding to the modals in the shared semantic space based on the semantic blocks. According to the method, the accuracy, generalization and stability of the data processing system in multiple tasks are guaranteed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a multi-level fine-grained semantic alignment method based on cross-modal information fusion. BACKGROUND

[0002] With the rich development of business system information carriers, data presents the characteristics of multi-modal, multi-source heterogeneous, multi-level expression, etc. Structured data such as databases and knowledge graphs, and unstructured data such as texts, images, audios, videos, etc. all exist at the same time. They differ significantly in semantic granularity, expression method, and time sequence relationship, which seriously affects the effectiveness of information fusion, intelligent understanding, and knowledge reasoning. Especially in the semantic alignment process, there are problems of semantic redundancy, conflict, and inconsistency between different modalities, causing understanding deviation and result error.

[0003] Most of the existing methods only support shallow modal splicing or only perform coarse-grained alignment at a fixed level (such as entity level), which is difficult to realize dynamic granularity semantic understanding and cannot meet the demand of multi-granularity expression and deep-level information extraction in actual application.

[0004] Existing methods are difficult to handle the significant differences in expression method and structural characteristics of multi-source heterogeneous data, especially when processing text, image, speech and other data at the same time, lacking a unified data preprocessing and modal coordination mechanism, resulting in inconsistent structural expression between modalities, which seriously affects the fusion efficiency.

[0005] Current mainstream semantic representation methods often use single-modal encoding structure or shallow feature splicing, which is difficult to capture deep semantic dependency relationships between modalities, resulting in distorted semantic space expression after fusion, affecting the performance of the model in high-order cognitive tasks. Therefore, how to design a fusion multi-model encoding mechanism to enable different modalities to have structural alignment capability and interactive modeling capability in the semantic space has become a difficult point that needs to be broken through.

[0006] Existing methods lack an extensible and evolvable semantic space construction mechanism, especially when the labeled data is scarce, the model lacks self-supervised optimization and structure enhancement capability, resulting in limited semantic consistency alignment effect, and it is difficult to form a high-quality shared semantic representation.

[0007] Therefore, there is an urgent need for a semantic alignment method that supports multi-modal fusion and hierarchical abstraction capability from bottom-level semantic representation to high-level semantic reasoning to ensure the accuracy, generalization and stability of the data processing system in multi-task. SUMMARY

[0008] The application provides a multi-level fine-grained semantic alignment method based on cross-modal information fusion to solve the above problems.

[0009] The first aspect of the application provides a multi-level fine-grained semantic alignment method based on cross-modal information fusion, comprising: Step S1: The multi-modal data preprocessing module pre-processes the obtained multi-modal input to generate a multi-modal intermediate expression format corresponding to the multi-modal input, and the multi-modal input includes text, image, voice and video; Step S2: input the multi-modal intermediate expression format into the encoder corresponding to each modality to obtain the embedding representation vector corresponding to each modality, map the embedding representation vector corresponding to each modality to a shared semantic space by a unified projection module, and form a feature expression corresponding to each modality in the shared semantic space; Step S3: divide the feature expression corresponding to each modality into a minimum semantic unit, determine the semantic feature granularity according to the task to be executed, combine the semantic similarity, the spatial and temporal proximity relationship and the cross-modal mapping rule to aggregate the minimum semantic unit to form a semantic block, and perform feature alignment and feature fusion on the feature expression corresponding to each modality in the shared semantic space based on the semantic block.

[0010] Preferably, the step S1: the multi-modal data preprocessing module pre-processes the obtained multi-modal input to generate a multi-modal intermediate expression format corresponding to the multi-modal input, comprising: Step S11: The multi-modal data preprocessing module performs format uniform processing on the text, and the format uniform processing includes line break processing, encoding format cleaning, paragraph structure recognition and punctuation repair; the processed text is divided into sentences and tagged with parts of speech, and based on the pre-defined semantic unit extraction rule, the key words, domain entities and logical relationships required for semantic alignment are identified; the processed text is divided into semantic blocks according to the semantic consistency, each semantic block is provided with a unique semantic fragment number and a label; and an intermediate expression format of the text is output, which includes the original text, the syntax structure, the keyword list, the semantic fragment number and the label corresponding to each block. Step S12: The multi-modal data preprocessing module performs pre-processing operation on the image, and the pre-processing operation includes image resolution correction, format conversion and standard size cropping; target detection is performed on the pre-processed image to determine the confidence of each detected target, and the image region corresponding to the frame of the target with a confidence exceeding a preset threshold is taken as the image semantic unit of the image; a visual semantic embedding vector and a natural language description segment are generated for each image semantic unit using a graphic-text pre-training model; and an intermediate expression format of the image is output, which is a hierarchical structure organized by a plurality of image semantic blocks, and the image semantic block includes an image semantic unit index, an image semantic unit, a position of the image semantic unit, a semantic label, OCR content, a visual semantic embedding vector and a natural language description segment. Step S13: The multimodal data preprocessing module performs preprocessing operations on the speech, including speech transcription, sentence segmentation, noise filtering, and emotion labeling, each retained sentence segment corresponds to a semantic segment; output the intermediate expression format of the speech, the intermediate expression format of the speech is a list of semantic segments organized in sequence, each semantic segment includes a transcription text, an emotion label, a frame-level timing information of speech speed, and a confidence score; Step S14: The multimodal data preprocessing module performs preprocessing operations on the video, including extracting and dividing the shot paragraphs, extracting key frames, audio, and subtitles from the shot paragraphs; identifying visual semantics for the key frames, performing speech transcription and emotion labeling on the audio corresponding to the shot paragraphs, and constructing video-speech-text triples according to the timestamps and frame indexes; aligning the subtitle text with the video timeline, and performing semantic segmentation on the text of the subtitles to construct a mapping matrix representing the correspondence between the key frames and the subtitle text segments; output the intermediate expression format of the video, the intermediate expression format of the video is a plurality of encapsulated video segments, each video segment includes a shot number, a key frame image, a subtitle text, a transcription content corresponding to the subtitle text, a semantic segment candidate list corresponding to the video segment, frame index and timestamp information; wherein each unit in the semantic segment candidate list is jointly constituted by the key frame image in the video segment corresponding to the unit and the subtitle segment and the audio segment within the time window corresponding to the unit.

[0011] Preferably, the predefined semantic unit extraction rule combines a structure perception and a semantic self-consistency mechanism, wherein the structure perception refers to automatically identifying a subject-predicate-object core path based on a syntax dependency tree to determine the start and end boundaries of a semantic segment; the semantic self-consistency introduces a semantic embedding similarity judgment within a sliding window, so that the semantic expression within the same semantic segment remains consistent.

[0012] Preferably, the step S2: inputting the multimodal intermediate expression format into the encoder corresponding to each modality to obtain the embedding representation vector corresponding to each modality, mapping the embedding representation vector corresponding to each modality to a shared semantic space by a unified projection module to form the feature expression corresponding to each modality, including: Step S21: constructing a shared semantic space; Step S22: the text encoding module performs embedding representation on the text intermediate expression format to obtain the text encoding corresponding to the text intermediate expression format, and maps it to the shared semantic space, the mapped result is used as the first text feature, and the first text feature is used as the feature expression corresponding to the text; Step S23: The image encoding module encodes each image semantic unit in the image intermediate expression format respectively; the position-aware multi-head attention mechanism is used to combine the spatial topological structure corresponding to the image intermediate expression format and the semantic correlation information, to obtain the inter-region semantic graph structure representation corresponding to the image intermediate expression format; based on the inter-region semantic graph structure representation, the cross-modal semantic embedding compatible with the text modality is generated for each image semantic unit and mapped to the shared semantic space, and the mapped result is taken as the first image feature, and the first image feature is taken as the feature expression corresponding to the image; Step S24: The speech encoding module performs semantic embedding on each semantic segment in the speech intermediate expression format, fuses the emotional information corresponding to the emotion label, the frame-level timing information of the speech speed, and the converted text, constructs the speech embedding representation corresponding to each semantic segment through the multi-modal enhancement encoding network, and maps to the shared semantic space, and the mapped result is taken as the first speech feature, and the first speech feature is taken as the feature expression corresponding to the speech; Step S25: The video encoding module extracts the text of the corresponding subtitle segment of each unit in the semantic segment candidate list of the video segment according to the time sequence; For each unit: the timestamp information of the unit is combined to synchronize and fuse the key frame image and the text of the subtitle segment corresponding to the unit; the image encoding module extracts the key frame image feature corresponding to the unit, and jointly models the text feature of the subtitle segment corresponding to the unit, to obtain the fusion semantic representation of the unit; The fusion semantic representations of each unit in the semantic segment candidate list of the video segment are weightedly fused to generate the fusion semantic representation of the video segment; The fusion semantic representations of all video segments are fused to obtain the fusion semantic representation corresponding to the video, and the fusion semantic representation corresponding to the video is mapped to the shared semantic space, and the mapped result is taken as the first video feature, and the first video feature is taken as the feature expression corresponding to the video; The dimensions of the first text feature, the first image feature, the first speech feature and the first video feature are the same.

[0013] Preferably, in the step S3, the feature expression corresponding to each modality is divided into minimum semantic units, including: The semantic block is taken as the minimum unit of the text, the image semantic block is taken as the minimum unit of the image, the semantic segment in the audio is taken as the minimum unit of the speech, and the video segment in the video is taken as the minimum unit of the video; based on the minimum unit corresponding to each modality, the feature expression corresponding to each modality is divided into minimum semantic units.

[0014] Preferably, in the step S3, the semantic feature granularity is determined according to the task to be executed, the minimum semantic units are aggregated to form semantic blocks in combination with semantic similarity, spatio-temporal proximity relationship and cross-modal mapping rules; and the feature alignment and feature fusion are performed on the feature expressions corresponding to each modality in the shared semantic space based on the semantic blocks, including: The task guiding module determines the semantic feature granularity according to the semantic target type or context complexity index of the task to be executed. The semantic aggregator adopts a hierarchical clustering algorithm or a multi-head attention mechanism to aggregate the minimum semantic units in the shared semantic space that are adjacent or similar to each other to a degree greater than a preset threshold to form cross-modal semantic blocks meeting the semantic feature granularity requirement. The feature alignment and feature fusion are performed on the first text feature, the first image feature, the first speech feature and the first video feature in the shared semantic space based on the cross-modal semantic blocks.

[0015] The second aspect of the present application provides a multi-level fine-grained semantic alignment device based on cross-modal information fusion, which comprises: A multi-modal data preprocessing module is configured to preprocess the obtained multi-modal input to generate a multi-modal intermediate expression format corresponding to the multi-modal input, and the multi-modal input includes text, image, speech and video. An encoding module is configured to input the multi-modal intermediate expression format into an encoder corresponding to each modality to obtain an embedding representation vector corresponding to each modality, and map the embedding representation vector corresponding to each modality to a shared semantic space by a unified projection module to form a feature expression corresponding to each modality. A fusion module is configured to divide the feature expression corresponding to each modality into minimum semantic units, determine the semantic feature granularity according to the task to be executed, aggregate the minimum semantic units in combination with semantic similarity, spatio-temporal proximity relationship and cross-modal mapping rules to form semantic blocks, and perform feature alignment and feature fusion on the feature expressions corresponding to each modality in the shared semantic space based on the semantic blocks.

[0016] The third aspect of the present application provides an electronic device, which comprises: At least one processor; and A memory connected in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0017] The fourth aspect of the present application provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method as described above.

[0018] The application provides a multi-level fine-grained semantic alignment method based on cross-modal information fusion.

[0019] The application has the following technical effects: (1) The application can realize multi-level and fine-grained semantic alignment. The application introduces a multi-modal data preprocessing mechanism, which can realize the semantic standardized expression of structured and unstructured data, and significantly improves the structural consistency and time alignment quality between modes. Especially in the scene involving multiple modalities such as text, image and voice, the application can effectively reduce the preprocessing complexity and improve the data availability.

[0020] (2) The application fuses a multi-model data encoding mechanism, which maintains the independence of modal features while establishing a semantic coordination relationship between modes through shared semantic embedding mapping and multi-head cross attention mechanism. This mechanism balances the cross-modal representation consistency and semantic expression diversity, improving the representation ability and generalization ability of the model.

[0021] (3) The application realizes dynamic fusion of multi-granularity semantics by constructing a shared semantic space and introducing a multi-level semantic alignment strategy. With the cooperation of contrast learning and self-supervised mechanism, even in the absence of labeled samples, the application can maintain high-precision semantic aggregation ability, enhancing the robustness and expansion ability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of the multi-level fine-grained semantic alignment method based on cross-modal information fusion provided by the application is provided. Figure 2 A general framework diagram of the multi-level fine-grained semantic alignment method based on cross-modal information fusion provided by the application is provided. Figure 3 A flowchart of multi-modal data preprocessing provided by the application is provided. Figure 4 A flowchart of data encoding of multi-model fusion is provided. Figure 5 A flowchart of semantic alignment based on a shared semantic space is provided. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] like Figures 1-2 As shown, a multi-level fine-grained semantic alignment method based on cross-modal information fusion is proposed, the method comprising: Step S1: The multimodal data preprocessing module preprocesses the acquired multimodal inputs to generate multimodal intermediate representation formats corresponding to the multimodal inputs. Multimodal data includes text, images, speech, and video. Step S2: Input the multimodal intermediate representation format into the encoder corresponding to each modality to obtain the embedding representation vector corresponding to each modality. The unified projection module maps the embedding representation vector corresponding to each modality to the shared semantic space to form the feature representation corresponding to each modality. Step S3: Divide the feature representations corresponding to each modality into the smallest semantic units, determine the semantic feature granularity according to the task to be executed, and aggregate the smallest semantic units by combining semantic similarity, spatiotemporal proximity and cross-modal mapping rules to form semantic blocks; perform feature alignment and feature fusion on the feature representations corresponding to each modality in the shared semantic space based on the semantic blocks.

[0025] This invention is aimed at multimodal data fusion scenarios in complex business operations. In the overall architecture design, a "multimodal data preprocessing module" is first established. This module is responsible for format standardization, structure extraction and semantic annotation of the original input text, image, voice and video data, so as to establish a unified data foundation and alignment interface for subsequent feature encoding and semantic alignment.

[0026] The main goal of the multimodal data preprocessing module is to construct an intermediate representation format that is structurally consistent, alignable, and easy to encode, while preserving the original semantic features of different modalities, so as to ensure that each modality has clear semantic boundaries and identifiable structural information before entering the fusion module.

[0027] Step S1: The multimodal data preprocessing module preprocesses the acquired multimodal input to generate a multimodal intermediate representation format corresponding to the multimodal input, including: Step S11: The multimodal data preprocessing module performs format unification processing on the text, including line break processing, encoding format cleaning, paragraph structure recognition, and punctuation repair; it then segments the processed text into sentences and performs part-of-speech tagging, identifying keywords, domain entities, and logical relationships required for semantic alignment based on predefined semantic unit extraction rules; it further segments the processed text into semantically consistent blocks, assigning a unique semantic segment number and label to each block; and finally, it outputs an intermediate text expression format, which includes the original text, syntactic structure, keyword list, and semantic segment numbers and labels corresponding to each block. Step S12: The multimodal data preprocessing module performs preprocessing operations on the image, including image resolution correction, format conversion, and standard size cropping; it performs target detection on the preprocessed image, determines the confidence level of each detected target, and uses the image region corresponding to the border of the target with a confidence level exceeding a preset threshold as the image semantic unit; it uses a pre-trained image-text model to generate a visual semantic embedding vector and a natural language description fragment for each image semantic unit; and it outputs an intermediate image representation format, which consists of several image semantic blocks organized in a hierarchical structure. Each image semantic block includes an image semantic unit index, an image semantic unit, the position of the image semantic unit, a semantic label, OCR content, a visual semantic embedding vector, and a natural language description fragment. Step S13: The multimodal data preprocessing module performs preprocessing operations on the speech, including speech transcription, sentence segmentation, noise filtering, and emotion labeling. Each retained sentence segment corresponds to a semantic segment. The output speech intermediate expression format is a sequentially organized list of semantic segments. Each semantic segment includes the transcribed text, emotion label, speech rate frame-level temporal information, and confidence score. Step S14: The multimodal data preprocessing module performs preprocessing operations on the video, including extracting and dividing shot segments, extracting keyframes, audio, and subtitles from the shot segments; identifying visual semantics for keyframes, performing speech-to-text transcription and emotion annotation on the audio corresponding to the shot segments, and constructing video-speech-text triples based on timestamps and frame indices; aligning the subtitle text with the video timeline, and semantically segmenting the subtitle text, constructing a mapping matrix representing the correspondence between keyframes and subtitle text segments; and outputting an intermediate video representation format, which consists of multiple encapsulated video segments. Each video segment includes a shot number, keyframe image, subtitle text, the transcribed content corresponding to the subtitle text, a semantic segment candidate list corresponding to the video segment, frame index, and timestamp information. Each unit in the semantic segment candidate list is composed of the keyframe image in that segment and the subtitle and audio segments within that time window. This information is used for subsequent multimodal semantic consistency alignment and aggregation operations.

[0028] Furthermore, the predefined semantic unit extraction rules integrate a dual mechanism of structure awareness and semantic self-consistency. Structure awareness refers to automatically identifying the subject-verb-object core path based on the syntactic dependency tree, thereby determining the start and end boundaries of the semantic segment. Semantic self-consistency introduces semantic embedding similarity judgment within the sliding window to ensure that the semantic expression within the same semantic segment remains consistent.

[0029] In this invention, the predefined semantic unit extraction rules support domain task adaptation and can automatically adjust the segmentation scale according to different contexts, thereby improving the consistency and accuracy of semantic fragment generation and cross-modal alignment.

[0030] Step S11 performs semantic consistency segmentation on the processed text, and sets a unique semantic segment number and label for each segment, so that semantic granularity can be distinguished as needed during encoding.

[0031] As a fundamental modality in multimodal information systems, text processing primarily revolves around semantic information extraction and fragment structure segmentation. After text structure cleaning, the semantic fragment recognition process begins. This process employs a rule-based and model-driven approach. First, general language processing tools are used for sentence segmentation and part-of-speech tagging to preliminarily identify the subject, verbs, objects, and modifiers in the text. Subsequently, a custom "semantic unit extraction rule set" is introduced to identify keywords, domain entities, logical relationships, and other information required for alignment with business semantics.

[0032] To improve the accuracy of subsequent cross-modal semantic alignment, the system divides sentences into semantically consistent blocks during text processing. Each block is assigned a unique semantic segment number and a corresponding label, allowing for differentiation of semantic granularity during encoding. The final output is stored in a structured format, including the original text, syntactic structure, keyword list, segment labels, and block IDs, for direct use by the encoder module.

[0033] In this invention, step S12 focuses on the extraction and semantic annotation of key visual regions. The input image data is first subjected to image resolution correction, format conversion (e.g., JPEG to PNG), and standard-size cropping to ensure that the input image meets the input specifications for subsequent model inference.

[0034] The core process relies on object detection models (such as YOLOv5, Faster R-CNN, and DETR) to detect major objects in an image. The model output includes the bounding box coordinates, class label, and confidence score for each detected object. The system retains only high-confidence regions (e.g., ≥0.5) and uses these regions as "image semantic units".

[0035] Next, the system introduces a pre-trained image-text model (such as CLIP or BLIP) to generate visual semantic embedding vectors and natural language description fragments for each detected region. For regions in the image that do not belong to a known category but have semantic meaning, the system supports the use of OCR to complete the visual annotation information.

[0036] The final structured representation of the image consists of several image semantic blocks. Each block contains an image region index, location information, semantic labels, OCR content, and a corresponding semantic vector, which are used for subsequent alignment with text fragments. Multiple regions in the image can form an image hierarchy to support the modeling of relative semantic relationships within the image.

[0037] In this invention, the main processing flow of voice data in step S13 includes four steps: speech transcription, sentence segmentation, noise filtering, and emotion annotation.

[0038] After receiving user voice input or audio tracks from a video, the system calls on an edge speech recognition model (such as DeepSpeech or wav2vec 2.0) to convert the raw audio stream into readable text information with timestamps. During the transcription process, the system records the start and end times and confidence scores of each word, and performs automatic sentence segmentation and paragraphing based on pause patterns.

[0039] To address the issue of decreased speech recognition accuracy in noisy environments, this invention introduces a "confidence filtering mechanism" based on a dual-gated filtering network: any speech segment with a sentence-level recognition confidence score below a preset threshold (such as below 0.85) will be automatically removed or marked as "pending confirmation" to avoid interference from noisy corpora with alignment quality.

[0040] In addition, to be compatible with multimodal emotion recognition tasks, the system can call a speech emotion analysis model to identify changes in pitch and intensity of speech, and mark the emotion type and intensity value of each speech segment. This information can be used as an auxiliary signal in subsequent semantic segment alignment and multimodal inference.

[0041] After preprocessing, the system organizes the speech content into a list of semantic segments. Each segment contains fields such as transcribed text, emotion tag, speech rate index, frame-level temporal information, and confidence score, and is aligned with the text modal structure.

[0042] In this invention, in step S14, the video modality, as a composite of image, audio, and subtitle information, requires a preprocessing flow that coordinates the timeline alignment and semantic extraction capabilities of the multimodal content. The system employs a three-step design for processing the original video content: keyframe extraction, audio analysis, and subtitle synchronization.

[0043] First, a shot boundary detection algorithm (such as SBD based on visual similarity) is used to divide the video into several shot segments. Within each shot segment, one or two representative frames are selected as "keyframes" using strategies such as image change rate, object density, and semantic distribution. These keyframes will represent the main visual semantics of the video segment in subsequent view encoding.

[0044] Secondly, the video audio track is extracted as an independent audio file, and the speech data processing module is called to perform speech transcription and emotion recognition operations. Based on the timestamp and frame index, a three-element temporal alignment relationship of "video-speech-text" is constructed.

[0045] For videos with external subtitles, the system can parse the subtitle text and automatically align it with the video timeline, constructing a "subtitle segment - keyframe" mapping matrix. In cases without subtitles or with a high subtitle error rate, the system defaults to using the speech-to-text transcription as the language modality basis.

[0046] After preprocessing, each video segment is encapsulated into a unified structure, including: shot number, keyframe image, subtitle text, transcribed content, semantic segment list, frame index, and timestamp information. All video data is ultimately converted into a standard "timeline-driven modal fusion unit," providing high-quality input for multimodal coding and semantic alignment.

[0047] Step S2: Input the multimodal intermediate representation format into the encoder corresponding to each modality to obtain the embedding representation vector corresponding to each modality. The unified projection module maps the embedding representation vector corresponding to each modality to the shared semantic space to form the feature representation corresponding to each modality, including: Step S21: Construct a shared semantic space; Step S22: The text encoding module embeds the intermediate text representation format to obtain the text encoding corresponding to the intermediate text representation format, and maps it to the shared semantic space. The mapped result is used as the first text feature, and the first text feature is used as the feature representation of the text. Step S23: The image encoding module encodes each semantic unit of the intermediate image representation format; a position-aware multi-head attention mechanism is used to combine the spatial topology and semantic relevance information corresponding to the intermediate image representation format to obtain the inter-region semantic graph structure representation corresponding to the intermediate image representation format; based on the inter-region semantic graph structure representation, a cross-modal semantic embedding compatible with the text modality is generated for each semantic unit of the image and mapped to the shared semantic space. The mapped result is used as the first image feature, and the first image feature is used as the feature representation of the image. Step S24: The speech coding module performs semantic embedding on each semantic segment in the intermediate speech expression format, integrates the emotional information corresponding to the emotion tag, the speech rate frame-level temporal information and the transposed text, constructs the speech embedding representation corresponding to each semantic segment through a multimodal augmented coding network, and maps it to the shared semantic space. The mapped result is used as the first speech feature, and the first speech feature is used as the feature expression corresponding to the speech. Step S25: The video encoding module extracts the text of each unit and its corresponding subtitle segment from the semantic segment candidate list of the video segment according to the time sequence; For each unit: the keyframe image and subtitle fragment text corresponding to the unit are synchronously fused by combining the timestamp information of the unit; the image encoding module extracts the keyframe image features corresponding to the unit and performs joint modeling with the text features of the subtitle fragment text corresponding to the unit to obtain the fused semantic representation of the unit; The fusion semantic representation of each unit in the candidate list of semantic segments of the video segment is weighted and fused to generate the fusion semantic representation of the video segment; The fusion semantic representations of all video segments are fused to obtain the fusion semantic representation of the video. The fusion semantic representation of the video is then mapped to the shared semantic space. The mapped result is used as the first video feature, and the first video feature is used as the feature expression of the video. Among them, the first text feature, the first image feature, the first speech feature, and the first video feature have the same dimension.

[0048] The intermediate video representation format consists of multiple encapsulated video segments. Each video segment includes a shot number, keyframe image, subtitle text, the transcribed content of the subtitle text, a semantic fragment candidate list corresponding to the video segment, frame index, and timestamp information. Each unit in the semantic fragment candidate list is composed of the keyframe image from that segment, along with the subtitle and audio fragments within that unit's corresponding time window. This is used for subsequent multimodal semantic consistency alignment and aggregation operations.

[0049] In this invention, after completing the standardized preprocessing of multimodal data, the data encoding module that integrates multiple models embeds and represents the input data of different modalities such as text, image, speech and video, extracts their deep semantic features, and maps the features of each modality to a shared semantic space, providing a basic representation for the subsequent semantic alignment module.

[0050] In this invention, the text encoding module in step S21 aims to perform deep semantic modeling on the input natural language text and extract contextual semantic vectors that can be used by the subsequent semantic alignment module. This invention uses a pre-trained Chinese language model based on the Transformer structure as the encoding backbone, such as mainstream models like RoBERTa-wwm-ext or BERT-chinese, and adapts it to multimodal task scenarios through fine-tuning.

[0051] First, the intermediate text representation is segmented using a tokenizer, mapped to a token sequence, and then converted into corresponding word vectors for input into the model. After multi-layer Transformer encoding, the model outputs contextual semantic information for each token. To obtain an aggregated representation of the entire text, the system selects the vector corresponding to the [CLS] position as the default text segment representation, or aggregates all token embeddings using average pooling according to task requirements.

[0052] The encoded text vectors are typically 768-dimensional. To ensure a unified representation with other modalities in the subsequent shared semantic space, the system uses a fully connected layer to project them to a unified dimension (e.g., 512-dimensional) and performs normalization to facilitate cross-modal similarity alignment. Furthermore, the text encoding module supports long text slice input. The system can segment text by sentence or paragraph and generate an embedding representation for each slice, adapting to the semantic distribution features of long texts.

[0053] In this invention, the image encoding module in step S22 is used to convert static images or image regions into embedding vectors with semantic representation capabilities. To enhance cross-modal fusion performance, this invention employs a Visual Transformer (ViT) as the backbone for image feature extraction and integrates a lightweight multi-object encoding network using the CLIP visual encoding strategy to independently encode visual features for each semantic unit (such as an object detection region). This network receives image regions cropped by the object detection model as input and extracts context-aware semantic vectors through linear embedding and a Transformer structure. For image patches containing OCR information, the system jointly inputs the visual embedding and OCR text embedding into the encoder to achieve joint image-text semantic modeling. The final output is a high-dimensional embedding representation of each semantic unit in the intermediate image representation format, retaining the region ID and location information, which serves as input for subsequent multimodal alignment.

[0054] In multi-region image scenarios, such as those from OCR detection boxes and object detection regions, the system supports encoding each image region separately and fusing region features into image segment-level semantic vectors through an attention weighting mechanism. After encoding, the image vectors are also mapped to the same shared dimensional space as the text through a linear layer, ensuring structural consistency during modality alignment.

[0055] In this invention, the speech encoding module in step S23 is responsible for converting the input audio data into a fixed-dimensional semantic vector representation. This invention uses the wav2vec 2.0 speech model as the encoding backbone. This model directly models at the original waveform level, possessing strong contextual representation capabilities, and is particularly suitable for processing unstructured speech content. The model outputs a frame-level feature sequence, which is then further weighted and aggregated based on information such as frame energy distribution, model confidence, or sentiment peaks, or adaptively calculated using an attention mechanism to generate a fixed-dimensional semantic representation of the segment.

[0056] Consistent with other modalities, the speech embedding is also normalized to a fixed dimension (e.g., 512-dimensional) after output through a mapping network to enter the shared semantic space. For multi-sentence continuous speech scenarios, the system allows for continuous processing using a sliding window mechanism, retains timestamp information, and supports synchronization alignment with video clips.

[0057] The video encoding module in step S24 needs to simultaneously integrate image, audio, and subtitle information to perform multi-channel semantic modeling on the input video segment. This invention breaks down video feature extraction into three channels: keyframe image encoding, audio track semantic extraction, and subtitle text modeling, and then fuses them at the semantic layer.

[0058] Keyframes are extracted at equal intervals along the timeline or based on an event detection algorithm by the video preprocessing module and then fed into the image encoding module for semantic extraction. Audio extraction is performed in the same way as the speech encoding module, preserving time synchronization features. If the video contains subtitles, these can be used as text input to the text encoding module to generate embedded subtitles.

[0059] The output vectors from the three channels are then input into a fusion network for weighted aggregation, forming a unified video segment-level semantic representation. The fusion mechanism employs an attention weighting mechanism to automatically adjust the contribution strength of each channel, thereby enhancing the overall semantic integrity of the video. The final video representation vector is also projected onto a shared semantic space, enabling it to participate in semantic alignment.

[0060] After step S25, the method further includes: Step S26: Normalize the first text feature, first image feature, first speech feature, and video feature after projection.

[0061] The shared semantic space proposed in this invention aims to provide a unified representation platform for data encoding results from different modalities, enabling semantically consistent aggregation of the same semantic content expressed by each modality within the space. In practical deployment, the shared semantic space is constructed through a trained and optimized vector space with a unified 512-dimensional dimension, serving as the mapping target for all modal embedding vectors.

[0062] The space is established based on the idea of ​​contrastive learning. The system is trained by constructing modality-paired samples, so that vector pairs from different modalities but expressing the same semantics have high similarity in the space, while vector pairs expressing different semantics maintain distance. Specifically, the optimization objective adopts the NT-Xent loss function (Normalized Temperature-scaled Cross Entropy Loss) and combines it with a dynamic sampling strategy to improve training efficiency and convergence speed.

[0063] To achieve consistent alignment of semantic embeddings across different modalities, this invention establishes a shared semantic space projection module after each modality encoding module. This module primarily uses a linear transformation network to map the original modality output embeddings to a vector space of the same dimension (e.g., 512-dimensional) and performs standardization to ensure the stability of similarity calculations.

[0064] The shared semantic space introduces a modality contrastive learning objective during training, aiming to make the projection results of the same semantic content as close as possible across different modalities, while keeping the embeddings of different semantics as far apart as possible. This process is constrained by the NT-Xent loss function and supplemented by a modality masking strategy to improve the model's robustness to missing modalities.

[0065] Furthermore, to enhance the discriminative power and spatial structure of the representation, the projection module incorporates modal feature normalization and dropout regularization mechanisms, effectively preventing overfitting during model training. The trained shared semantic space exhibits high distribution consistency and semantic aggregation capabilities, forming the core foundation for the subsequent semantic alignment module to construct the semantic relationship graph.

[0066] In step S3, the feature representations corresponding to each modality are segmented into the smallest semantic units, including: Semantic blocks are used as the smallest unit of text, image semantic blocks as the smallest unit of image, semantic segments in audio as the smallest unit of speech, and video segments in video as the smallest unit of video. Based on the smallest unit corresponding to each modality, the feature representation corresponding to each modality is segmented into the smallest semantic unit.

[0067] In step S3, the semantic feature granularity is determined according to the task to be executed. Combining semantic similarity, spatiotemporal proximity, and cross-modal mapping rules, the smallest semantic units are aggregated to form semantic blocks. Based on these semantic blocks, feature alignment and feature fusion are performed on the feature representations corresponding to each modality in the shared semantic space, including: The task guidance module determines the semantic feature granularity based on the semantic target type or contextual complexity index of the task to be executed. The semantic aggregator uses a hierarchical clustering algorithm or a multi-head attention mechanism to aggregate the smallest semantic units in the shared semantic space that are adjacent or have a similarity greater than a preset threshold to form cross-modal semantic blocks that meet the requirements of semantic feature granularity. Based on cross-modal semantic blocks, feature alignment and feature fusion are performed on the first text feature, first image feature, first speech feature, and first video feature in the shared semantic space.

[0068] In this invention, the semantic segmentation is further deconstructed into multiple "semantic units" according to the granularity of semantic segmentation. Each unit is the smallest semantic expression fragment with complete contextual information. The text modality usually uses sentences or phrases as semantic units, the image modality uses detection box regions or visual patches as basic units, the speech modality uses semantically stable duration as the basic window generation unit, and the video modality uses shots or event segments as the unit basis.

[0069] While constructing semantic units, the system retains meta-information such as their source modality, timestamp, and contextual association, forming a set of semantic fragments. Each fragment is fed into a unified projection layer and mapped to a shared semantic space for subsequent semantic alignment calculations.

[0070] Considering the varying semantic granularity requirements of different tasks, this invention introduces a dynamic granularity adjustment mechanism during the alignment process. This mechanism consists of two parts: a semantic aggregator and a task guidance module. The semantic aggregator uses hierarchical clustering algorithms or multi-head attention mechanisms to aggregate adjacent or similar semantic segments into semantic blocks; the task guidance module determines the required semantic granularity level based on external task signals (such as classification, reasoning, and retrieval).

[0071] For example, in image and text retrieval tasks, the system can retain the smallest granular semantic fragments to achieve precise point-to-point alignment; while in event summary generation tasks, the system can aggregate multiple semantic fragments into high-level semantic concepts, reducing alignment details and improving efficiency.

[0072] In the shared semantic space, semantic units across modalities are aligned using semantic similarity calculation. Specifically, the dot product similarity calculation formula is used: Here, a and b represent two embedding vectors from different modalities. In actual implementation, the system uses a batch matrix calculation method to construct a similarity matrix between all segments and sets a similarity threshold to determine "alignment pairs".

[0073] This invention employs a local matching mechanism, whereby each target segment selects only the most relevant modal segments as alignment candidates, avoiding information redundancy and computational overhead associated with fully connected components. By introducing a dynamic threshold filtering mechanism, the system can control segment alignment density while maintaining high matching accuracy.

[0074] After alignment, cross-modal semantic fusion is performed based on the similarity and modal importance between segments. The fusion method employs a weighted average mechanism, combined with a modal attention factor, to determine the proportion of different modalities in the fusion result. The generated fusion vector serves as the final semantic unit representation, possessing high-quality semantic aggregation capabilities and can be invoked by downstream tasks.

[0075] This invention also supports adding labels, relational structure information, and alignment confidence scores to the fusion results to assist in deeper semantic applications such as question-answering reasoning and relation extraction.

[0076] A multi-level fine-grained semantic alignment device based on cross-modal information fusion, the device comprising: Multimodal data preprocessing module: configured to preprocess the acquired multimodal inputs and generate multimodal intermediate representation formats corresponding to the multimodal inputs. Multimodal data includes text, images, audio, and video. Encoding module: Configured to input the multimodal intermediate representation format into the encoder corresponding to each modality, obtain the embedding representation vector corresponding to each modality, and then the unified projection module maps the embedding representation vector corresponding to each modality to the shared semantic space to form the feature representation corresponding to each modality; Fusion module: It is configured to divide the feature representations corresponding to each modality into the smallest semantic units, determine the semantic feature granularity according to the task to be executed, and aggregate the smallest semantic units by combining semantic similarity, spatiotemporal proximity and cross-modal mapping rules to form semantic blocks; based on the semantic blocks, feature alignment and feature fusion are performed on the feature representations corresponding to each modality in the shared semantic space.

[0077] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0078] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0079] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections may include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections may include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0080] It should be noted that these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Furthermore, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Additionally, these modules can be integrated together to form a System-on-a-Chip (SOC).

[0081] The electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0082] The present invention also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, is used to perform the above-described method embodiments.

[0083] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0086] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multi-level fine-grained semantic alignment method based on cross-modal information fusion, characterized in that, The methods include: Step S1: The multimodal data preprocessing module preprocesses the acquired multimodal inputs to generate multimodal intermediate representation formats corresponding to the multimodal inputs. Multimodal data includes text, images, speech, and video. Step S2: Input the multimodal intermediate representation format into the encoder corresponding to each modality to obtain the embedding representation vector corresponding to each modality. The unified projection module maps the embedding representation vector corresponding to each modality to the shared semantic space to form the feature representation corresponding to each modality. Step S3: Divide the feature representations corresponding to each modality into the smallest semantic units, determine the semantic feature granularity according to the task to be executed, and aggregate the smallest semantic units by combining semantic similarity, spatiotemporal proximity and cross-modal mapping rules to form semantic blocks; perform feature alignment and feature fusion on the feature representations corresponding to each modality in the shared semantic space based on the semantic blocks.

2. The method as described in claim 1, characterized in that, Step S1: The multimodal data preprocessing module preprocesses the acquired multimodal input to generate a multimodal intermediate representation format corresponding to the multimodal input, including: Step S11: The multimodal data preprocessing module performs format unification processing on the text, including line break processing, encoding format cleaning, paragraph structure recognition, and punctuation repair; it then segments the processed text into sentences and performs part-of-speech tagging, identifying keywords, domain entities, and logical relationships required for semantic alignment based on predefined semantic unit extraction rules; it further segments the processed text into semantically consistent blocks, assigning a unique semantic segment number and label to each block; and finally, it outputs an intermediate text expression format, which includes the original text, syntactic structure, keyword list, and semantic segment numbers and labels corresponding to each block. Step S12: The multimodal data preprocessing module performs preprocessing operations on the image, including image resolution correction, format conversion, and standard size cropping; it performs target detection on the preprocessed image, determines the confidence level of each detected target, and uses the image region corresponding to the border of the target with a confidence level exceeding a preset threshold as the image semantic unit; it uses a pre-trained image-text model to generate a visual semantic embedding vector and a natural language description fragment for each image semantic unit; and it outputs an intermediate image representation format, which consists of several image semantic blocks organized in a hierarchical structure. Each image semantic block includes an image semantic unit index, an image semantic unit, the position of the image semantic unit, a semantic label, OCR content, a visual semantic embedding vector, and a natural language description fragment. Step S13: The multimodal data preprocessing module performs preprocessing operations on the speech, including speech transcription, sentence segmentation, noise filtering, and emotion labeling. Each retained sentence segment corresponds to a semantic segment. The output speech intermediate expression format is a sequentially organized list of semantic segments. Each semantic segment includes the transcribed text, emotion label, speech rate frame-level temporal information, and confidence score. Step S14: The multimodal data preprocessing module performs preprocessing operations on the video, including extracting and dividing shot segments, extracting keyframes, audio, and subtitles from the shot segments; identifying visual semantics for keyframes, performing speech-to-text transcription and emotion annotation on the audio corresponding to the shot segments, and constructing video-speech-text triples based on timestamps and frame indices; aligning the subtitle text with the video timeline, and semantically segmenting the subtitle text, constructing a mapping matrix representing the correspondence between keyframes and subtitle text segments; and outputting the intermediate video representation format, which consists of multiple encapsulated video segments. Each video segment includes a shot number, keyframe image, subtitle text, the transcribed content corresponding to the subtitle text, a semantic segment candidate list corresponding to the video segment, frame index, and timestamp information. Each unit in the semantic segment candidate list is composed of the keyframe image in the corresponding video segment and the subtitle and audio segments within the corresponding time window.

3. The method as described in claim 2, characterized in that, The predefined semantic unit extraction rules integrate a dual mechanism of structure awareness and semantic self-consistency. Structure awareness refers to automatically identifying the subject-verb-object core path based on the syntactic dependency tree, thereby determining the start and end boundaries of the semantic segment. Semantic self-consistency introduces semantic embedding similarity judgment within the sliding window to ensure that the semantic expression within the same semantic segment remains consistent.

4. The method as described in claim 1, characterized in that, Step S2: Input the multimodal intermediate representation format into the encoder corresponding to each modality to obtain the embedding representation vector corresponding to each modality. The unified projection module maps the embedding representation vector corresponding to each modality to the shared semantic space to form the feature representation corresponding to each modality, including: Step S21: Construct a shared semantic space; Step S22: The text encoding module embeds the intermediate text representation format to obtain the text encoding corresponding to the intermediate text representation format, and maps it to the shared semantic space. The mapped result is used as the first text feature, and the first text feature is used as the feature representation of the text. Step S23: The image encoding module encodes each semantic unit of the intermediate image representation format; a position-aware multi-head attention mechanism is used to combine the spatial topology and semantic relevance information corresponding to the intermediate image representation format to obtain the inter-region semantic graph structure representation corresponding to the intermediate image representation format; based on the inter-region semantic graph structure representation, a cross-modal semantic embedding compatible with the text modality is generated for each semantic unit of the image and mapped to the shared semantic space. The mapped result is used as the first image feature, and the first image feature is used as the feature representation of the image. Step S24: The speech coding module performs semantic embedding on each semantic segment in the intermediate speech expression format, integrates the emotional information corresponding to the emotion tag, the speech rate frame-level temporal information and the transposed text, constructs the speech embedding representation corresponding to each semantic segment through a multimodal augmented coding network, and maps it to the shared semantic space. The mapped result is used as the first speech feature, and the first speech feature is used as the feature expression corresponding to the speech. Step S25: The video encoding module extracts the text of each unit and its corresponding subtitle segment from the semantic segment candidate list of the video segment according to the time sequence; For each unit: the keyframe image and subtitle fragment text corresponding to the unit are synchronously fused by combining the timestamp information of the unit; the image encoding module extracts the keyframe image features corresponding to the unit and performs joint modeling with the text features of the subtitle fragment text corresponding to the unit to obtain the fused semantic representation of the unit; The fusion semantic representation of each unit in the candidate list of semantic segments of the video segment is weighted and fused to generate the fusion semantic representation of the video segment; The fusion semantic representations of all video segments are fused to obtain the fusion semantic representation of the video. The fusion semantic representation of the video is then mapped to the shared semantic space. The mapped result is used as the first video feature, and the first video feature is used as the feature expression of the video. Among them, the first text feature, the first image feature, the first speech feature, and the first video feature have the same dimension.

5. The method as described in claim 4, characterized in that, In step S3, the feature representations corresponding to each modality are segmented into the smallest semantic units, including: Semantic blocks are used as the smallest unit of text, image semantic blocks as the smallest unit of image, semantic segments in audio as the smallest unit of speech, and video segments in video as the smallest unit of video. Based on the smallest unit corresponding to each modality, the feature representation corresponding to each modality is segmented into the smallest semantic unit.

6. The method as described in claim 5, characterized in that, In step S3, the semantic feature granularity is determined according to the task to be executed. Combining semantic similarity, spatiotemporal proximity, and cross-modal mapping rules, the smallest semantic units are aggregated to form semantic blocks. Based on these semantic blocks, feature alignment and feature fusion are performed on the feature representations corresponding to each modality in the shared semantic space, including: The task guidance module determines the semantic feature granularity based on the semantic target type or contextual complexity index of the task to be executed. The semantic aggregator uses a hierarchical clustering algorithm or a multi-head attention mechanism to aggregate the smallest semantic units in the shared semantic space that are adjacent or have a similarity greater than a preset threshold to form cross-modal semantic blocks that meet the requirements of semantic feature granularity. Based on cross-modal semantic blocks, feature alignment and feature fusion are performed on the first text feature, first image feature, first speech feature, and first video feature in the shared semantic space.

7. A multi-level fine-grained semantic alignment device based on cross-modal information fusion, characterized in that, The device includes: Multimodal data preprocessing module: configured to preprocess the acquired multimodal inputs and generate multimodal intermediate representation formats corresponding to the multimodal inputs. Multimodal data includes text, images, audio, and video. Encoding module: Configured to input the multimodal intermediate representation format into the encoder corresponding to each modality, obtain the embedding representation vector corresponding to each modality, and then the unified projection module maps the embedding representation vector corresponding to each modality to the shared semantic space to form the feature representation corresponding to each modality; Fusion module: It is configured to divide the feature representations corresponding to each modality into the smallest semantic units, determine the semantic feature granularity according to the task to be executed, and aggregate the smallest semantic units by combining semantic similarity, spatiotemporal proximity and cross-modal mapping rules to form semantic blocks; based on the semantic blocks, feature alignment and feature fusion are performed on the feature representations corresponding to each modality in the shared semantic space.

8. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1-6.

Images