Multi-modal file intelligent classification and label generation method and system

Abstract

The invention relates to the technical field of artificial intelligence and information processing, and particularly discloses a multi-modal file intelligent classification and label generation method and system. The method comprises the steps of obtaining text paragraphs, image segments and video key frame data in a file, performing format recognition and region separation, and generating multi-modal content structure data; semantic features are extracted based on the data, and fused semantic representation is constructed; generating a main label, a sub-label and a keyword set by using the fused semantic vector, and constructing a multi-level label structure; and further performing label redundancy elimination and structure optimization to generate a label atlas, and performing consistency analysis and feedback optimization through the label atlas and the semantic representation structure. Compared with the prior art, the method has the advantages that various modal information can be effectively fused, the accuracy, hierarchical structure and semantic consistency of archive label generation are improved, and the method has high intelligence, self-adaption and sustainable optimization capabilities and is suitable for various scenes such as archive management, content auditing and semantic archiving.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and information processing, and in particular to a multi-modal archive intelligent classification and label generation method and system. BACKGROUND

[0002] With the growing demand for digital archive management, more and more archive data exhibits multi-modal characteristics, covering complex forms such as structured and unstructured text passages, image segments, and video key frames. Traditional archive classification and label generation methods mainly rely on manual annotation or single-modal feature extraction methods, which are difficult to effectively cope with the heterogeneity and coupling of multi-modal archive content in the semantic dimension, resulting in low accuracy, poor semantic consistency, and lack of structured expression ability in label generation results, which seriously affects the archive retrieval efficiency and intelligent recommendation effect.

[0003] In the prior art, some studies have introduced deep learning models for semantic extraction of text or image content, but there are still many limitations in multi-modal fusion processing, especially in the following three aspects: (1) lack of unified fusion semantic representation structure, which cannot effectively model the context association of multi-modal features; (2) lack of label generation mechanism based on semantic hierarchy, which makes it difficult to realize multi-level and multi-granularity expression of labels; (3) lack of feedback optimization mechanism, the system cannot adjust the label model parameters according to the historical classification results, and it is difficult to adapt to semantic evolution or label system changes.

[0004] Therefore, there is an urgent need for an archive label generation method that can fuse multi-modal archive content such as text, images, and videos, construct a unified fusion semantic representation structure, and combine a label hierarchy generation model and a label graph structure to realize a label generation method with strong semantic consistency, clear classification hierarchy, and feedback optimization capability, to improve the automatic classification quality and structured archiving capability of multi-modal archives. SUMMARY

[0005] The present application provides a multi-modal archive intelligent classification and label generation method and system to solve the problem of how to construct a unified fusion semantic representation structure based on multi-modal input information such as text passages, image segments, and video key frames, and realize a multi-level, high-consistency, and optimizable feedback archive semantic label automatic generation method by introducing a hierarchical label generation model and a label graph structure.

[0006] To solve the above technical problems, the present application provides a multi-modal archive intelligent classification and label generation method, comprising:

[0007] Obtaining text passages, image segments, and video key frame data in the archive to be processed, generating multi-modal archive content structure data through format recognition, content area separation, and standardization processing;

[0008] Based on the multi-modal archive content structure data, semantic encoding processing, image region feature encoding and frame-level feature modeling are performed to extract text paragraph, image segment and video key frame feature vectors;

[0009] The text paragraph, image segment and video key frame feature vectors are subjected to modal alignment, cross-modal joint modeling and context attention extraction processing to generate a fusion semantic representation structure;

[0010] According to the fusion semantic representation structure, a main tag set, a sub-tag set and a keyword set are generated, hierarchical combination processing is performed, and a multi-level label structure data is constructed;

[0011] The label structure is realized by constructing a graph structure, and the multi-level label graph is defined as:

[0012]

[0013] Wherein, V (n) is a label node set; E (n) is a label semantic edge set;

[0014] The label importance is sorted based on the PageRank algorithm:

[0015]

[0016] Wherein, is the weight of label node i in the tth iteration; V (n) is a node set; δ is a jump probability; is the adjacent node set of label i;

[0017] The expression of the multi-level label structure is:

[0018]

[0019] Wherein, is the label graph structure of the nth sample; V (n) is a node set; E (n) is an edge set; π (n) is a node weight vector; μ is a Mobius function level mapping strength;

[0020] The semantic redundancy and label reconstruction processing are performed on the multi-level label structure data to generate an optimized label set and a label structure atlas;

[0021] Based on the label structure atlas and the fusion semantic representation structure, consistency analysis and update processing are performed to complete the feedback optimization of the cross-modal fusion model and the label generation model.

[0022] Further, the step of generating the multi-modal archive content structure data comprises:

[0023] Obtaining a preliminary analysis result through format recognition and content area separation operation;

[0024] Performing character recognition, target area detection and frame screening processing to obtain a content structure set;

[0025] Encoding and unifying the content structure set and standardizing the position information to generate the multi-modal archive content structure data.

[0026] Further, the steps of the text paragraph, image segment and video key frame feature vector further comprise:

[0027] Obtaining a text paragraph in the multi-modal archive content structure data, inputting the text semantic model for semantic encoding processing to generate a text paragraph feature vector;

[0028] Obtaining an image segment in the multi-modal archive content structure data, inputting the image semantic model for image area feature encoding to generate an image segment feature vector;

[0029] Obtaining a video key frame in the multi-modal archive content structure data, inputting the video semantic model for frame-level feature modeling to generate a video key frame feature vector.

[0030] Further, the text paragraph feature vector, image segment feature vector and video key frame feature vector are subjected to modal alignment processing to generate an initial alignment vector set, a cross-modal fusion model is constructed based on the initial alignment vector set, cross-modal joint modeling processing is performed, and context attention extraction processing is performed on the fusion semantic representation vector to generate a fusion semantic representation structure.

[0031] Further, the fusion semantic representation structure is obtained, inputting a label generation model to perform a primary label generation operation to generate a preliminary primary label set.

[0032] Further, based on the preliminary primary label set and the fusion semantic representation structure, label hierarchy decoding processing is performed to generate a sub-label and keyword set.

[0033] Further, the primary label set, the sub-label set and the keyword set are subjected to hierarchical combination processing to generate multi-level label structure data.

[0034] Further, the steps of generating the optimized label set and the label structure atlas further comprise:

[0035] Obtaining multi-level label structure data, inputting a label clustering model for semantic redundancy processing to obtain a label clustering structure;

[0036] The label clustering structure is subjected to label reconstruction processing to generate an optimized label set with clear semantic structure.

[0037] A label structure graph is generated based on the optimized label set to construct a semantic association path between labels.

[0038] Further, the step of completing feedback optimization of the cross-modal fusion model and the label generation model further comprises:

[0039] The label structure graph and the fusion semantic representation structure are obtained, and label coverage and consistency analysis processing is performed;

[0040] According to the analysis processing result, the attention parameters in the cross-modal fusion model and the label generation model are updated;

[0041] The updated attention parameters are applied to the fusion semantic representation structure generation process to complete model feedback optimization update.

[0042] A multi-modal archive intelligent classification and label generation system is applied to any of the above methods and comprises:

[0043] A content structure extraction module is configured to obtain text paragraphs, image segments and video key frame data, and generate multi-modal archive content structure data;

[0044] A feature extraction module is configured to extract text paragraph feature vectors, image segment feature vectors and video key frame feature vectors from the multi-modal archive content structure data;

[0045] A fusion modeling module is configured to perform modal alignment, cross-modal joint modeling and context attention extraction to generate a fusion semantic representation structure;

[0046] A label generation module is configured to generate multi-level label structure data based on the fusion semantic representation structure;

[0047] A label optimization module is configured to perform semantic redundancy removal and label reconstruction on the multi-level label structure data to generate a label structure graph;

[0048] A feedback optimization module is configured to update the attention parameters of the cross-modal fusion model and the label generation model based on the label structure graph and the fusion semantic representation structure;

[0049] A user interface and model management module provides graphical configuration interfaces, task management consoles and model parameter monitoring functions, and can upload archive data, view label generation results and export multi-level label structures through the interface.

[0050] The key innovations of the present application include:

[0051] (1) Multi-modal fusion semantic vector construction mechanism, through the adaptive attention mechanism, the text, image and video features are combined to realize the unified semantic representation modeling of cross-modal archive content.

[0052] (2) Hierarchical structure design of label generation, the semantic decoding path between main label, sub-label and keyword is constructed, and the superior-inferior and co-occurrence relationship between labels is modeled based on graph structure.

[0053] (3) Introducing the keyword selection mechanism of information entropy and semantic gradient, the information theory and derivative optimization tools are fused in the label refinement process, and the stability and generalization ability of label extraction are improved.

[0054] (4) Graph algorithm enhancement of label structure atlas, combining PageRank label sorting and Mobius function to construct superior-inferior mapping relationship, so that the generated label system has structural and calculable properties.

[0055] (5) Label-model feedback optimization mechanism, a consistency detection and feedback path between the fusion label result and the semantic representation vector is constructed to realize the adaptive and long-term evolution ability of the classification system.

[0056] The main beneficial effects are as follows:

[0057] (1) Unified semantic modeling of multi-modal archives is realized, and the accuracy of label expression and context understanding ability are improved. The text semantic encoding, image region feature modeling and video frame level abstract representation are jointly represented for the first time, and the importance of different modalities is dynamically weighted through the fusion attention mechanism to construct a fusion semantic representation vector. Compared with the traditional method which cannot effectively process heterogeneous modal content, the fusion structure can realize the consistency modeling of cross-modal semantics, and significantly improve the accuracy of label generation and the context relevance of expression.

[0058] (2) A hierarchical label generation mechanism is proposed to realize the semantic layering and structural archiving expression of labels. The main label generation module, self-label decoding module and keyword extraction module are combined with sub-label path modeling and label semantic structure graph construction to form a multi-level label structure. Especially in S420-S430, PageRank sorting and Mobius function are introduced to construct the label atlas structure, effectively capturing the semantic inclusion and dependency relationship between labels. This design greatly improves the hierarchical clarity and structured management ability of the archive label system, breaking through the limitations of existing methods which only support "label list type" single layer expression.

[0059] (3) Introducing differentiable information theory and gradient regularization mechanism to enhance the stability and generalization ability of label generation. In the keyword extraction process, the application innovatively introduces the joint optimization target of Shannon entropy function and gradient regularization term to construct a keyword screening expression, which screens key elements from the dimensions of information stability and semantic sensitivity, and improves the robustness and explainability of the label generation model to sample changes in actual scenarios.

[0060] (4) Constructing an optimizable feedback semantic consistency mechanism to promote continuous evolution of the model. Compared with the traditional one-time label prediction mode, the application dynamically analyzes the consistency between the label structure graph and the semantic fusion representation, and combines the model feedback optimization process (S600), so that the label generation model can be automatically corrected and self-optimized based on historical classification errors and prediction bias, and the ability of the model to adapt to long-term evolution scenarios (such as label set adjustment or new class appearance) is improved.

[0061] (5) Full-process mathematical structure closed loop to support transparent and traceable model reasoning chain. The application constructs a formula progressive transmission structure between modules, each variable is clearly defined, consistent in units, and logically closed, which facilitates traceability and mathematical stability verification of the system implementation process, and provides theoretical and engineering support for the system in high consistency requirement scenarios such as security archives, financial supervision, and medical archiving. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 A flowchart of a multi-modal archive intelligent classification and label generation method provided by an embodiment of the application;

[0063] Figure 2 A structural block diagram of a multi-modal archive intelligent classification and label generation system provided by an embodiment of the application. DETAILED DESCRIPTION

[0064] Embodiment one: reference Figure 1 is a flowchart of a multi-modal archive intelligent classification and label generation method provided by an embodiment of the application, which can at least include steps S100-S600:

[0065] S100, generating multi-modal archive content structure data through format recognition, content area separation and standardization processing.

[0066] S200, based on the multi-modal archive content structure data, performing semantic coding processing, image region feature coding and frame-level feature modeling to extract text paragraph, image segment and video key frame feature vectors.

[0067] S300, performing modal alignment, cross-modal joint modeling and context attention extraction processing on the text paragraph, image segment and video key frame feature vectors to generate a fusion semantic representation structure.

[0068] S400, generating a main tag set, a sub-tag set and a keyword set according to the fused semantic representation structure, performing hierarchical combination processing, and constructing a multi-level tag structure data.

[0069] S500, performing semantic redundancy elimination and tag reconstruction processing on the multi-level tag structure data, and generating an optimized tag set and a tag structure atlas.

[0070] S600, performing consistency analysis and update processing based on the tag structure atlas and the fused semantic representation structure, and completing feedback optimization of the cross-modal fusion model and the label generation model.

[0071] Step S100 includes at least steps S110-S130:

[0072] S110, obtaining text paragraphs, image segments and video key frame data in the to-be-processed archives, performing format recognition and content area separation, and obtaining preliminary analysis results.

[0073] Specifically, when implementing this step, the system first calls an input module to load the mixed format archive data submitted by the user, which includes but is not limited to heterogeneous format files such as PDF, DOCX, TIFF, MP4, JPG, etc. In order to be compatible with multiple source data and extract the processable content therein, the system performs format recognition processing, including three operations of format header detection, file type inference and content structure judgment. Among them, the format header detection is based on the file binary signature table to complete content identification analysis, and the file type inference combines MIME type and suffix verification method to confirm the archive format structure.

[0074] After completing the format recognition, the system performs content area separation processing on the classified archive data. For text files, the system uses a preset paragraph recognition model and line block clustering mechanism to mark out continuous semantic blocks to form independent text paragraph areas. For image content, the system divides the readable area of the image segment through image page edge contour detection and layout analysis algorithm, and performs layer de-stacking reconstruction processing on the multi-layer overlapping part to ensure that the image boundary extraction is clear and complete. For archive content containing video format, the system calls a video frame extraction module, uses scene change point detection method to divide the video content into time slices, extracts representative video key frame data, and forms a video sub-fragment index sequence.

[0075] The preliminary analysis results output by this step include three types of structured content, namely: text paragraph structure, image segment structure and video key frame structure, which will serve as the input data basis for subsequent content recognition and feature extraction.

[0076] S120, text recognition processing is performed on the text paragraphs in the preliminary analysis result, target region detection is performed on the image segments, and frame screening processing is performed on the video key frames to obtain a content structure set.

[0077] After obtaining the preliminary analysis result, the system performs corresponding processing operations on the text paragraph, image segment, and video key frame structure to further improve the semantic expression ability and structural integrity.

[0078] Specifically, for the text paragraph structure in the preliminary analysis result, the system calls an OCR recognition engine module to perform text recognition processing. The text recognition processing includes three stages of text positioning, layout restoration, and character classification. In the text positioning stage, a text bounding box detection model based on CTPN is used to mark each line of text in the paragraph; in the layout restoration stage, a convolution structure is used to extract the arrangement relationship within the line to restore the sentence structure; in the character classification stage, a pre-trained character recognition model is called to convert the raster image into a standard character stream and perform semantic splicing. Finally, a text semantic unit with clear structure levels is obtained.

[0079] Further, for the image segment structure, the system performs target region detection operations. The target region detection calls an image detection network based on Faster-RCNN to perform saliency detection and bounding box extraction on the image segment in the image-text mixed arrangement area, table area, handwritten annotation area, etc. The detection process includes region proposal generation, feature map extraction, and class recognition, and the output results include the class label, boundary coordinates, and confidence score of the detected region. The system aggregates all detected region results and establishes an index mapping relationship with the original image.

[0080] For the video key frame structure, the system performs frame screening processing using a joint screening mechanism based on image similarity and semantic sparsity. First, duplicate key frames are filtered through the Perceptual Hash algorithm; second, semantically similar frames are excluded through image semantic embedding distance calculation; finally, key frames with high change degree and information density are retained to generate frame screening results.

[0081] The system integrates the text paragraph recognition results, image target region recognition results, and screened video key frame results obtained after the above three processes into a content structure set, which serves as the basic input for subsequent standardization processing and semantic feature modeling.

[0082] S130, encoding unification and position information standardization processing are performed on the content structure set to generate a multi-modal archive content structure data.

[0083] Further, after obtaining the above content structure set, in order to guarantee the alignment consistency and coding adaptability of multi-modal information in the subsequent cross-modal fusion model, the system performs coding unification and position information standardization processing.

[0084] In terms of coding unification, the system first gives a unified content identification number to the text paragraph, image segment and video key frame according to the multi-modal unified description protocol. The number structure is generated based on a hierarchical index mechanism, including source file number, modal type identification, content position index and other fields, forming a "modal unified identification vector". Then, the system uses a unified content embedding module to perform format conversion and vector coding operation on the three types of data. The text paragraph is coded into a standardized text sequence structure; the image segment is regularized into a fixed size image matrix; the video key frame is serialized into an image frame set and attached with timestamp information.

[0085] In terms of position information standardization, the system maps the page number, intra-paragraph position of the text paragraph, page coordinates and resolution ratio of the image segment, timestamp and frame position sequence of the video key frame to a unified spatial coordinate system. A hierarchical mapping algorithm based on document structure tree is used to normalize the position information, ensuring that all content structures have retrievable and comparable positioning properties in subsequent model input.

[0086] The results of the coding unification processing and position information standardization processing form multi-modal archive content structure data with complete structure, clear semantics and unified numbering. The multi-modal archive content structure data will be used as the input source of S210, S220 and S230 in the subsequent feature extraction step, where the text paragraph structure data will be used as the input object of the text semantic coding model S210, the image segment structure data will be used as the input data of the image region feature coding model S220, and the video key frame structure data will be used as the basis sample of the frame-level feature modeling model S230.

[0087] By implementing the above steps, the system realizes the standardized extraction and normalization modeling of multi-modal archive content in the semantic, structural and positioning aspects, providing high-quality and unified input basis for the feature coding of subsequent text paragraphs, image segments and video key frames, thereby significantly improving the accuracy and consistency of subsequent label generation and classification modeling. The constructed multi-modal archive content structure data has the properties of alignability, aggregability and high expressiveness, and is the core prerequisite for the invention to form a semantic label system and multi-modal classification result.

[0088] Step S200 includes at least steps S210-S230:

[0089] S210, obtain the text paragraph in the multi-modal archive content structure data, input the text semantic model for semantic coding processing, and generate a text paragraph feature vector.

[0090] Specifically, when implementing this step, the system first calls the text paragraph reading interface to extract the standardized text paragraph data that has completed OCR recognition, position normalization, and format marking processing from the multi-modal archive content structure data. The text paragraph data is input as structured text with clear paragraph numbers, original page numbers, content start and end positions, and normalized semantic coding boundaries.

[0091] After completing the text paragraph acquisition, the system calls the text semantic coding module to load the preset text semantic model. The text semantic model is constructed based on context-aware language modeling technology and uses a multi-layer stacked bidirectional coding structure to handle semantic extension and syntactic dependency problems. The input of the text semantic model includes the standardized paragraph content and structure labels, including title indicators, body annotations, and note identifiers.

[0092] The text semantic model sequentially performs word segmentation, word embedding, context modeling, and output representation generation operations. The system encodes each semantic unit in the paragraph into a fixed-dimensional word vector through the word embedding module and constructs multiple layers of attention channels in the context modeling module to capture long-distance dependencies and inter-sentence logical relationships. Subsequently, the semantic representation generation module fuses and reduces the embedding vectors based on context weights, and finally outputs a paragraph semantic representation vector corresponding to the original paragraph.

[0093] The above paragraph semantic representation vector serves as the "text paragraph feature vector" and will be used in the subsequent S310 step of modal alignment operation, and will be jointly input into the cross-modal modeling module with the image segment feature vector and the video key frame feature vector.

[0094] S220, acquire image segments in the multi-modal archive content structure data, input image semantic model for image region feature coding, and generate image segment feature vectors.

[0095] Specifically, after completing the semantic modeling of the text paragraph in S210, the system enters the image segment feature extraction process. The system first calls the image segment set from the multi-modal archive content structure data to extract the image region information that has completed boundary box labeling and size normalization. The image segments are stored in matrix form and are accompanied by normalized position information, image page numbers, and image segment identification numbers for each segment to ensure their locatability in the document semantic system.

[0096] After image segments are input into the image semantic model, image preprocessing operations are first performed, including pixel normalization, size scaling, and channel normalization, to ensure that all image segments meet the input constraints of the model structure. Subsequently, the system loads the image semantic model and performs region feature encoding processing. The image semantic model adopts a multi-layer convolutional-pooling structure combination, with nested region interest extraction units to capture visual differences between the interior and boundaries of regions.

[0097] During the convolutional processing stage, the model performs filtering operations on image segments at multiple scales to extract spatial texture features at different levels. During the pooling stage, the model reduces the spatial size while retaining the main image structural information. Subsequently, the system calls the region interest module to perform candidate region proposal, bounding box rescoring, and region aggregation on the extracted intermediate feature maps, and outputs the embedded semantic feature vector of the image region.

[0098] The system performs a fusion operation on the feature vectors of multiple regions corresponding to each image segment to generate a unique image segment feature vector, maintaining a one-to-one correspondence with the original image segment identifier. This image segment feature vector, as input, will participate in the modality alignment process in subsequent step S310 along with the text segment feature vector and the video keyframe feature vector.

[0099] S230. Obtain video keyframes from the multimodal archive content structure data, input them into the video semantic model to perform frame-level feature modeling, and generate video keyframe feature vectors.

[0100] Furthermore, after completing image segment feature modeling in S220, the system enters the video keyframe feature modeling stage. The system retrieves a set of video keyframes from the multimodal archive content structure data. The set of keyframes is a representative set of image frames obtained in the preceding S120 through scene change point detection and frame filtering mechanism, and has frame position numbers, timestamp indexes, and intra-frame texture density labels.

[0101] The system inputs the video keyframes into a video semantic model to perform frame-level feature modeling. The video semantic model adopts a time-aware structure, embedding local inter-frame similarity modeling units and cross-frame semantic accumulation modules. To improve the accuracy of frame-level feature modeling, the system performs serialization padding and time interval normalization on the keyframes to form a stable frame sequence input.

[0102] The video semantic model extracts single-frame structural features through a shallow intra-frame visual perception network and constructs an inter-frame relationship graph in a deep structure to model the contextual dependencies of keyframes in the temporal domain. At each time step, the model outputs the embedding vector of the current frame and reduces the output vectors of all keyframes using a global attention mechanism to generate frame-level feature semantic vectors.

[0103] Each key frame feature vector output by the system finally outputs the frame number, timestamp and attribution segment index, and is used as one of the alignment input vectors of the video modality in S310, and is subsequently combined with the text passage feature vector and the image segment feature vector for cross-modal joint modeling.

[0104] Through the implementation of this step, the system respectively completes the context semantic modeling of the text passage, the region-level feature encoding of the image segment and the frame-level time sequence modeling of the video key frame based on the multi-modal archive content structure data, forms three types of feature vector sets with high expression ability, and constructs a unified input structure for the subsequent modal alignment and semantic fusion link. This design guarantees the consistency and semantic aggregation of cross-modal information coding, effectively supports the label generation algorithm based on the dynamic attention mechanism in the present application, and realizes the continuous mapping process from the structure data to the semantic space.

[0105] Step S300 at least includes steps S310-S330:

[0106] S310, the text passage feature vector, the image segment feature vector and the video key frame feature vector are subjected to modal alignment processing to generate an initial alignment vector set.

[0107] Specifically, the step S310 establishes a unified representation structure based on the text passage feature vector generated in step S210, the image segment feature vector generated in step S220 and the video key frame feature vector generated in step S230. The text passage feature vector is obtained by performing semantic encoding processing on the text passage in the multi-modal archive content structure data by a text semantic model, and is represented as a vector sequence with strong association with context semantics. The image segment feature vector is derived from the result of performing image region feature encoding on the multi-modal archive image segment in an image semantic model, and has spatial layout sensitivity and semantic distribution stability. The video key frame feature vector is a structural feature output obtained by performing frame-level feature modeling on the filtered key frame in a video semantic model, and has time sequence continuity and action scene expression ability.

[0108] To realize the unified semantic space mapping between the three types of modal feature vectors, the system first establishes a modal registration strategy, which performs feature alignment between heterogeneous modalities based on a structure mapping function and a semantic distribution matrix. Further, the semantic weight of each modality in the current archive structure is calculated through a multi-modal attention framework, and the modal consistency gating structure is used to dynamically adjust the feature alignment parameters of each modality. The above registration process finally generates an initial alignment vector set containing multi-source semantic information such as text, image and video.

[0109] The initial alignment vector set serves as the input basis for subsequent cross-modal joint modeling, and its internal structure has the following characteristics: first, the features of each modality form a collaborative space expression structure after being projected in the unified semantic dimension; second, the initial alignment vector set retains the structural identification of the original modality, ensuring that the subsequent processing can perform modality adaptive fusion; third, the interaction coefficient between vectors is automatically allocated by an attention gate network to achieve cross-modal redundant information suppression.

[0110] S320, constructing a cross-modal fusion model based on the initial alignment vector set, performing cross-modal joint modeling processing, and generating a fusion semantic representation vector.

[0111] After the construction of the initial alignment vector set is completed in step S310, step S320 performs deep fusion modeling processing on the vector set to generate a fusion semantic representation vector with unified semantic expression capability. The cross-modal fusion model is composed of multiple parallel structures, corresponding to the text modality fusion subnetwork, the image modality fusion subnetwork, and the video modality fusion subnetwork, and is integrated by a modality unified mapping structure.

[0112] Specifically, the system calls the cross-attention substructure to capture the semantic correlation between different modal fragments in the initial alignment vector set. The attention module adopts a hierarchical alignment mechanism, i.e., constructing attention calculation paths at the feature layer, the semantic layer, and the context layer. The feature layer alignment module mainly completes the fusion of different modal original information granularity, the semantic layer alignment module realizes the concept merging of modality label information, and the context layer alignment module is responsible for constructing the scene and intent association graph.

[0113] Further, the cross-modal fusion model introduces a modality compression mechanism to perform vector dimension reduction processing on redundant modal fragments, and adjusts the fusion threshold in combination with the context, to avoid a single modality having too high a proportion in the fusion output. The fusion semantic representation vector output by the model has the following characteristics: first, the fusion semantic representation vector has eliminated the modality boundary in the semantic space; second, the representation vector has balanced weights on the input multi-modal information, and can perform fine-grained decomposition according to subsequent task requirements.

[0114] S330, performing context attention extraction processing on the fusion semantic representation vector to generate a fusion semantic representation structure.

[0115] After the construction of the fusion semantic representation vector is completed in step S320, step S330 performs context attention extraction processing based on the fusion semantic representation vector to further generate a fusion semantic representation structure with structural indication.

[0116] Specifically, the system adopts a dynamic context sliding window mechanism to scan the fusion semantic representation vector, extracts feature fragments with high matching degree of semantic position, high-frequency vocabulary and image target area. The sliding window adjusts its width and step according to the document structure level to ensure adaptive processing capability for different archive types.

[0117] The system establishes the dependency path between semantic nodes through a structural context graph modeling method, updates the weight distribution in the dependency path by using a graph neural network module based on dynamic change of edge weight, and constructs an attention center graph pointing to the fusion semantic core area.

[0118] On the basis of the above graph, the system further introduces a semantic conflict detection module to identify sub-vectors with large meaning deviation or modal difference in the fusion semantic representation vector, and performs local clustering and reallocation through a semantic re-aggregation mechanism to complete the optimization construction of the fusion semantic representation structure.

[0119] The fusion semantic representation structure is composed of a multi-layer nested structure, which includes a top-level scene semantic dimension, a middle-level concept alignment dimension and a bottom-level modal encoding dimension. The three-layer structure ensures that the main label extraction, sub-label expansion and keyword extraction of the subsequent label generation module (step S410) have sufficient semantic basis.

[0120] Through the collaborative processing of steps S310, S320 and S330, the present application realizes the deep unified expression of heterogeneous feature data in multi-modal archives, and breaks through the technical bottlenecks of existing classification and label systems in modal isolation, fusion rigidity and context rupture. Specifically, the modal alignment processing realizes the semantic coordinate unification of multi-source information; the cross-modal joint modeling provides a general path for modal information fusion and semantic reconstruction; the context attention extraction completes the semantic focus pointing and structure optimization on the basis of fusion expression, providing accurate input for subsequent label generation and feedback optimization. This paragraph is the semantic modeling hub in the core technical route of the present application, and has significant technical progress characteristics.

[0121] Step S400 includes at least steps S410-S430:

[0122] S410, obtain the fusion semantic representation structure, input the label generation model to perform the main label generation operation, and generate a preliminary main label set.

[0123] This step first calls the fusion semantic representation vector generated in step S330. Let the fusion semantic vector of the nth archive sample be denoted as:

[0124]

[0125] Wherein:

[0126] is the fused semantic representation vector of the nth sample.

[0127] d: dimension of the fused vector.

[0128] is a d-dimensional real space.

[0129] n∈{1,2,…,N}: sample number, N is the total number of archive samples in the current batch.

[0130] The fused semantic vector is calculated by attention-weighted calculation of the feature vectors of multi-modal input (text, image, video):

[0131]

[0132] wherein:

[0133] are the text modality, image modality and video modality feature vectors of the nth sample, respectively;

[0134] are the attention weights of the three modalities, satisfying

[0135] is the modality projection weight matrix.

[0136] is the modality bias term.

[0137] The fused semantic representation vector is input into the main label generation model, and an autoregressive attention mechanism is adopted:

[0138]

[0139] wherein:

[0140] is the lth main label of the nth sample.

[0141] is the label sequence before the current label is generated.

[0142] is the query vector of the lth main label.

[0143] is the position-dependent attention scaling coefficient.

[0144] Attention(·): multi-head self-attention function.

[0145] softmax(·): normalization activation function.

[0146] The final output main label set is defined as:

[0147]

[0148] Wherein:

[0149] The main label set generated by the nth sample;

[0150] K n : represents the number of main labels generated by the nth sample.

[0151] S420, based on the main label set and the fusion semantic representation structure, the label hierarchy decoding process is performed to generate the sub-label and keyword set.

[0152] After obtaining the main label set , the system generates its corresponding sub-label based on each main label and semantic representation vector. The sub-label generation function adopts the conditional parallel path modeling form:

[0153]

[0154] Wherein:

[0155] represents the jth sub-label;

[0156] The attention weight of the qth path to the jth sub-label;

[0157] Q: the number of parallel paths;

[0158] Path weight and bias;

[0159] σ(·): Sigmoid activation function.

[0160] The output sub-label set is:

[0161]

[0162] Wherein:

[0163] The sub-label set generated by the nth profile sample;

[0164] The jth sub-label, where j∈{1,2,…,M n};

[0165] M n : the number of sub-labels generated by the nth sample, which is a sample-specific variable. The keyword set is selected based on semantic importance and stability indicators, and is defined as:

[0166]

[0167] wherein:

[0168] denotes the t-th keyword;

[0169] keyword candidate dictionary;

[0170] predicted probability of keyword k in sample n;

[0171] entropy function;

[0172] gradient of keyword probability pair fusion semantics;

[0173] γ: regularization term coefficient.

[0174] The final output keyword set is:

[0175]

[0176] wherein:

[0177] keyword set generated by the n-th profile sample;

[0178] the t-th keyword;

[0179] T n : total number of keywords generated by sample n.

[0180] S430, hierarchical combination processing is performed on the main label, sub-label and keyword set to generate multi-level label structure data.

[0181] The label structure is implemented by constructing a graph structure, and the multi-level label graph is defined as:

[0182]

[0183] wherein: label node set;

[0184] E (n) : label semantic edge set, established by co-occurrence relationship and semantic distance function.

[0185] Label importance is sorted based on the PageRank algorithm:

[0186]

[0187] wherein:

[0188] Weight of label node i in the tth round of iteration;

[0189] V (n) : Node set, consisting of main label, sub-label and keyword;

[0190] δ: Jump probability (generally 0.15);

[0191] Adjacent node set of label i.

[0192] The hierarchical relationship is calculated by the Mobius function to calculate the mapping strength between labels:

[0193]

[0194] Wherein: μ(i,j): The hierarchical mapping coefficient between label nodes i and j;

[0195] Combination number, indicating the number of ways to select k from j;

[0196] The function is used to express the inclusion and subordination relationship of labels in the combined label scene, and satisfies the hierarchical nature of the label graph structure.

[0197] The finally constructed label structure output is:

[0198]

[0199] Wherein:

[0200] Label graph structure of the nth sample;

[0201] V (n) : Node set, consisting of main label, sub-label and keyword;

[0202] E (n) : Edge set, representing the semantic, superior or co-occurrence connection between labels;

[0203] π (n) : Node weight vector, calculated by PageRank;

[0204] μ: Mobius function hierarchical mapping strength.

[0205] Through the collaborative processing of steps S410, S420 and S430, the application constructs a multi-level label generation and graph modeling method based on fusion semantic representation vector, realizes the precise classification of multi-modal archive semantic structure and the construction of high-consistency label system, and breaks through the key bottlenecks such as "modality fragmentation, label hierarchy missing and structure semantic disconnection" of the existing archive intelligent label generation system.

[0206] Specifically, step S410 is based on the fusion semantic representation vector Introducing modal attention mechanism and main label autoregressive generation model, completing the abstract compression of multi-modal semantic elements and the extraction of main label set Ensure that the label generation has overall semantic consistency and dynamicity of modal weight distribution.

[0207] Step S420 introduces a conditional path parallel model, which decodes the sub-label set with subordinate structure under the joint semantic constraint of the fusion vector and the main label And combined with gradient stability and semantic entropy criterion to filter the semantic keyword set Significantly improve the delicacy of label granularity and the controllability of semantic focus, solve the problem of random sub-label generation and redundant keyword selection in traditional methods.

[0208] Step S430 constructs a unified label graph structure Fuse the PageRank iteration ordering result π (n) And the Moebius function level mapping strength μ, realize the structure dependence modeling and semantic propagation optimization among multi-layer labels. This structure provides accurate structure input for subsequent redundancy processing and consistency feedback optimization of label graph, supports weight adjustment, level correction and semantic path reconstruction between labels.

[0209] This step provides a highly integrated and interpretable technical solution in terms of fusion of expression-driven main label generation, label path parallel decoding, multi-scale keyword filtering and graph-based semantic structure construction, effectively solves the problems of semantic fragmentation, label structure flatness and semantic redundancy in the construction of multi-modal archive label system, and has significant technical progress and wide engineering applicability. The structural label output not only optimizes the efficiency of archive information archiving, but also provides structural basis support for label visualization, label consistency retrieval and subsequent feedback optimization module, and is the core hub of the closed-loop semantic modeling system of the application.

[0210] Step 500 at least contains steps S510-S530:

[0211] S510, acquire multi-level label structure data, input label clustering model for semantic redundancy reduction, and obtain label clustering structure.

[0212] Specifically, the system first acquires the multi-level label structure data constructed in step S430, which includes a main label set, a sub-label set, and a keyword set. Each label entity is automatically derived by a label generation model supported by a fused semantic representation structure, and has a high degree of semantic distribution density under the cross support of different modal information. However, due to the dynamic distribution characteristics of the attention mechanism in the label generation model, some labels with similar semantics or repeated contexts may exist in the structure at different granularities, resulting in ambiguous semantic expression boundaries, and further adversely affecting subsequent classification, retrieval, and feedback learning.

[0213] The system inputs the above multi-level label structure data into a label clustering model for semantic redundancy reduction processing. The label clustering model can be implemented based on a K-means variant model with cluster center migration, a graph structure spectral clustering model, or a density estimation model based on BERT semantic space embedding. The goal is to merge labels with a semantic distance less than a certain similarity threshold and generate a label semantic clustering structure. The clustering structure expresses the semantic class center with a cluster center vector as the core, expresses the adjacent label expansion set with a boundary vector set, records the nesting and inclusion relationship between labels, and forms a complete label clustering semantic representation structure.

[0214] Further, the clustering process is not only based on the text semantic representation vector space, but also fuses the label activation patterns corresponding to the image segments and video key frames to ensure the consistency of the semantic clustering structure in the multi-modal context. After completing the initial screening of the clustering, the system will also use the context attention mechanism to assist in judging the boundaries of synonymous and near-synonymous labels in the context to improve the clustering accuracy.

[0215] To avoid the deletion of key labels or distortion of label centers due to semantic averaging, the system uses an explainable mechanism to score and evaluate the formation process of each cluster center, and only retains label compression units with high semantic compression rate and low information loss rate to enter the next stage.

[0216] S520, label reconstruction processing is performed on the label clustering structure to generate an optimized label set with clear semantic structure.

[0217] After completing the label clustering structure construction in step S510, the system enters the label semantic reconstruction processing phase. The reconstruction process aims to perform semantic reconstruction processing on the original label set according to the clustered label structure, and then generate an optimized label set with clear label boundaries, clear and separable upper and lower levels, and minimal semantic redundancy.

[0218] Specifically, the system first extracts the semantic main path of each clustering unit, i.e., the semantic main direction of the label semantic aggregation, which is established through a nested relationship tree structure, and truncates the path in combination with the context sequence boundary provided by the aforementioned fusion semantic representation structure to form a label chain hierarchical structure within the cluster. On this basis, the system performs semantic migration processing on the boundary labels in the original label set that deviate greatly from the cluster center, and rebinds them to the nearest label main path branch to form a semantic mapping relationship matrix of multi-class labels.

[0219] Further, the system performs a label reconstruction operation based on the constructed semantic mapping relationship matrix. The operation redivides and reorders the label set according to the semantic transfer probability and hierarchical position difference between labels, eliminates label entries with high semantic overlap overlap rate, and supplements possible missing secondary labels through a candidate label expansion mechanism. In the above operation, the system further references the logical structure between the initial main label and the sub-label to ensure that the newly generated optimized label set still maintains the order and interpretability of the hierarchical structure.

[0220] Understandably, in this phase, the system can also introduce modal attention features from the fusion semantic representation structure as reference indicators to ensure that the label reconstruction process fully reflects the multi-modal representation features between text, image, and video key frames, further improving the rationality of label semantic aggregation.

[0221] The final output of the optimized label set will have the following characteristics: there is no obvious redundancy or semantic conflict between labels, the hierarchical structure is clear, the semantic direction is stable, and it can completely cover the core semantic units of the input archive.

[0222] S530, generating a label structure map based on the optimized label set, and constructing a semantic association path between labels.

[0223] After completing the reconstruction of the optimized label set, the system enters the label structure map construction phase. The map not only provides structural display between labels, but also establishes logical reasoning relationships between label units through the construction of semantic association paths, providing structural support for subsequent model feedback optimization and human-computer interactive label assisted generation.

[0224] Specifically, the system first acquires the optimized label set output in step S520, and constructs an initial label graph structure relying on the hierarchical attribute information and semantic similarity information in the label set. The graph structure includes a node set and an edge set, the node set corresponds to each semantic label item, and the edge set is used to express the inclusive relationship, synonymous path relationship, and theme attachment relationship between labels. Edge weights can be calculated by the distance of labels in the semantic space, or automatically annotated by historical label co-occurrence frequency.

[0225] Further, to improve the dynamic adaptability of the label structure graph, the system introduces an inter-label context transition factor in the graph construction process, records the attention trajectory in the fused semantic representation structure, deduces the dynamic jump path between labels, and forms a dynamic updating mechanism of the semantic path graph. This mechanism has a significant advantage when there is a difference between multi-modal representations (for example, a semantic that appears in the video modality but is missing in the text modality). It can automatically complete the label path and ensure the closure of the label chain structure.

[0226] After generating the label structure graph, the system also performs topological sparsification processing on the graph, retains key main paths and high semantic contribution paths, and eliminates interference edges and redundant edges to ensure that the graph structure is simple and efficient. In addition, to improve the retrieval efficiency and scalability of the graph, the system stores the label structure graph in the form of a graph database, supports subsequent multi-task semantic reasoning, semantic annotation assisted generation, and user interactive label correction.

[0227] Further, the label structure graph generated in this step not only realizes hierarchical semantic organization between label sets, but also constructs reasoning paths between semantics, effectively supporting cross-modal label unified management and model consistency analysis requirements.

[0228] Through the semantic redundancy reduction and label reconstruction processing mechanism of this step S500, the application effectively alleviates the problems of label repetition, label boundary ambiguity, and label upper and lower level conflict under multi-modal fusion. Especially after introducing the label clustering model and the label reconstruction processing mechanism, the system can extract the most representative and most recognizable label combination from the complex label candidate set, and further organize the semantic relationship through the label structure graph to realize the unified expression of label information in the spatial structure and semantic topology.

[0229] Compared with the existing label generation mechanism which only relies on the output of the static decoder to output the label sequence, the application realizes the "label redundancy reduction-label reconstruction-label graph construction" three-in-one closed-loop control mode in the multi-modal label generation task through the continuous processing chain of S510 to S530, effectively improves the semantic accuracy, logical consistency and expansion ability of the label system, and has high engineering application value and promotion prospect.

[0230] Step S600 includes at least steps S610-S630:

[0231] S610, obtain the label structure graph and the fused semantic representation structure, and perform label coverage and consistency analysis processing.

[0232] Specifically, the system first acquires the label structure graph generated by step S530. The label structure graph is a directed graph structure constructed on the basis of the optimized label set, which contains the semantic association path between labels, the hierarchical pointing relationship, and the inheritance mapping relationship of superior and inferior labels. The structure is composed of label nodes in the optimized label set, semantic path edges between label nodes, and label semantic similarity information carried by each edge, forming a complete label semantic topology network.

[0233] Further, the system synchronously acquires the fusion semantic representation structure output by step S330. The fusion semantic representation structure is a set of embedded semantic vectors after fusing the text paragraph features, image segment features, and video key frame features, in which the modal alignment results, context attention weights, and semantic aggregation information have been embedded, which can fully express the representation features of the target archive in the multi-modal semantic space.

[0234] After the above data structure acquisition is completed, the system performs label coverage analysis processing. Label coverage analysis calculates the matching degree between each hierarchical label in the label structure graph and the corresponding semantic representation in the fusion semantic representation structure to determine whether the fusion semantic space effectively covers the semantic requirements of all label nodes in the label graph. If a label node cannot find sufficient semantic support or context alignment evidence in the fusion representation, it is determined that there is a lack of semantic coverage. Further, the system calculates the current label coverage index according to the ratio of the number of label nodes that can be explained by the fusion representation to the total number of label nodes.

[0235] Based on the label coverage analysis, the system further carries out label consistency analysis processing. The consistency analysis refers to the consistency matching degree between the superior-inferior path relationship in the label structure graph and the context attention path in the actual fusion semantic representation structure. The system constructs the actual semantic transmission path between labels, identifies the label aggregation trend implied in the fusion semantic structure by analyzing the attention matrix and context focusing behavior in the fusion semantic structure, and compares it with the semantic inheritance path defined in the label structure graph to obtain the consistency performance of the current model in the label hierarchical structure modeling.

[0236] Further, the system outputs two core evaluation indicators: the first is the label coverage value, and the second is the semantic consistency score. These two indicators will be used as key input parameters in the model updating stage to guide the correction direction of subsequent attention parameters and the focus area of model structure adjustment.

[0237] S620, according to the analysis processing result, update the attention parameters in the cross-modal fusion model and the label generation model.

[0238] Based on the label coverage and consistency analysis results output in step S610, the system enters the feedback adjustment process of the attention mechanism parameters. This process acts on the cross-modal fusion model and the label generation model two subsystems respectively, and updates the attention weight distribution strategy, modal attention mechanism and label level scheduling mechanism inside them.

[0239] Specifically, for the cross-modal fusion model, the system adjusts the dynamic weight coefficients of different modal features according to the label coverage results. In the region corresponding to the modal with low label coverage, the system enhances the attention degree of the modal feature vector in the fusion model; on the contrary, in the modal with high label redundancy or large semantic ambiguity rate, the system appropriately compresses the weight of the modal to prevent it from interfering with the fusion results. The above operation is realized by a dynamic weight regulation module, and its input is the difference mapping relationship between the modal source distribution recorded in the fusion semantic representation structure and the label coverage distribution in the label structure graph.

[0240] Further, for the label generation model, the system updates the level weight coefficients in the level attention extraction mechanism. If it is found in the label consistency analysis that the attention response of a label level in the fusion representation significantly deviates from the expected path in the label structure graph, the system reassigns the decoding attention parameters of each level in the label generation model through the feedback mechanism, thereby improving the modeling accuracy of the label semantic path in the label generation process. The update operation is completed by a label level decoding weight regulation module, and the attention parameter matrix is reconstructed according to the actual path difference matrix.

[0241] Further, the system also refines the adjustment of the context attention extraction strategy in the fusion semantic structure to ensure that the model can dynamically adapt to changes in content distribution density when processing multi-modal mixed documents and improve the label reasoning ability in the case of modal boundary ambiguity. The updated model parameters are stored in the model parameter mapping table structure for subsequent fusion and label generation task calls.

[0242] S630, apply the updated attention parameters to the fusion semantic representation structure generation process to complete the feedback optimization update of the model.

[0243] After completing the attention parameter update, the system re-injects the above update results into the generation process of the fusion semantic representation structure as input, and performs feedback optimization processing. The feedback optimization process includes three core tasks of fusion model reconstruction, label path regeneration and context attention reinforcement.

[0244] The system calls the updated cross-modal fusion parameter set on the basis of the original multi-modal feature input unchanged, re-executes the modal alignment, joint modeling and context extraction operation. This process can be understood as adjusting the internal modal fusion path and semantic convergence mode of the model structure without changing the model structure, thereby outputting a new set of fusion semantic representation structure. The structure will be better than the previous round output in terms of semantic coverage and path consistency of the label structure graph.

[0245] The system re-executes the label generation process based on the updated fusion semantic representation structure, including three steps of main label generation, sub-label generation and keyword extraction, and reconstructs the multi-level label set structure. The above label regeneration process significantly enhances the explanation consistency between the label and the fusion semantic, avoiding label error attribution caused by context deviation or modal ambiguity.

[0246] The system injects the update result into the context attention extraction mechanism, dynamically adjusts the attention window size and stepping strategy, so that the model can adaptively scale the attention domain according to the attention distribution when processing long text paragraphs, complex image regions or redundant frame sequences, improve the attention allocation efficiency and expression density. The final output optimized semantic structure can be used for the next round of label reasoning and feedback update, realizing the closed-loop evolution of the model level.

[0247] Through the complete process of S610 to S630, the system not only realizes the bidirectional consistency evaluation based on the label graph and the fusion semantic structure, but also constructs an automatic optimization closed-loop mechanism based on the attention parameter feedback. Compared with the traditional static label generation method, the method of the application introduces a dynamic label coverage rate analysis and path consistency regulation mechanism, so that the label generation process can adaptively respond to semantic missing and modal conflict problems, significantly improving the classification accuracy and label explanation of multi-modal archive processing.

[0248] The key innovations of the application include:

[0249] (1) Multi-modal fusion semantic vector construction mechanism, which realizes unified semantic representation modeling of cross-modal archive content through adaptive attention mechanism combined with text, image and video features.

[0250] (2) Hierarchical structure design of label generation, which innovatively constructs the semantic decoding path among main label, sub-label and keyword, and models the superior-inferior and co-occurrence relationship between labels based on graph structure.

[0251] (3) Introducing the keyword selection mechanism of information entropy and semantic gradient, which integrates information theory and derivative optimization tools in the label refinement process, improving the stability and generalization ability of label extraction.

[0252] (4) The graph algorithm of the label structure map is enhanced, the PageRank label sorting is combined with the construction of the Möbius function to construct an up-down mapping relationship, so that the generated label system has structural and calculable properties.

[0253] (5) A label-model feedback optimization mechanism is constructed to build a consistency detection and feedback path between the label results and the semantic representation vector, so that the classification system can adapt and evolve in the long term.

[0254] The following are its main beneficial effects:

[0255] (1) Unified semantic modeling of multi-modal archives is realized, and the accuracy of label expression and the ability of context understanding are improved. The present application first jointly represents text semantic coding, image region feature modeling and video frame-level abstract representation, and dynamically weights the importance of different modalities through the fusion attention mechanism to construct a fusion semantic representation vector. Compared with the traditional method which cannot effectively process heterogeneous modal content, the fusion structure can realize cross-modal semantic consistency modeling, significantly improving the accuracy of label generation and the context relevance of expression.

[0256] (2) A hierarchical label generation mechanism is proposed to realize semantic layering and structural archiving expression of labels. The present application forms a multi-level label structure through a main label generation module, a self-label decoding module and a keyword extraction module, combined with sub-label path modeling and label semantic structure graph construction. Especially in S420-S430, the PageRank sorting and the Möbius function are introduced to construct the label map structure, effectively capturing the semantic inclusion and dependency relationship between labels. This design greatly improves the hierarchical clarity and structured management ability of the archive label system, breaking through the limitations of existing methods which only support single-layer "label list" expression.

[0257] (3) Differentiable information theory and gradient regularization mechanism are introduced to enhance the stability and generalization ability of label generation. In the keyword extraction process, the present application innovatively introduces a joint optimization target of Shannon entropy function and gradient regularization term to construct a keyword screening expression, which screens key elements from two dimensions of information stability and semantic sensitivity, improving the robustness and explainability of the label generation model to sample changes in actual scenarios.

[0258] (4) A semantic consistency mechanism that can be optimized and fed back is constructed to promote the continuous evolution of the model. Compared with the traditional one-time label prediction mode, the present application dynamically analyzes the consistency between the label structure map and the semantic fusion representation, combined with the model feedback optimization process (S600), so that the label generation model can be automatically corrected and self-optimized based on historical classification errors and prediction biases, improving the ability of the model to adapt to long-term evolution scenarios (such as label set adjustment or appearance of new categories).

[0259] (5) Full-process mathematical structure closed loop, support model reasoning chain transparent traceable. The application constructs a formula progressive transmission structure between modules, each variable definition is clear, unit is consistent, logic is closed, which is convenient for system to realize process traceable and mathematical stability verification, and provides theoretical and engineering guarantee for system in high consistency requirement scene such as safety archives, financial supervision, medical archiving and the like.

[0260] Embodiment two: Figure 2 The structural block diagram of the multi-modal archive intelligent classification and label generation system according to the embodiment of the application is shown. As shown in the figure, Figure 2 The structure can include:

[0261] The content structure extraction module 10 is used for format analysis and content area separation on the input multi-modal archive, and completes the standardized conversion of original information. Specifically, it includes extracting text paragraphs, image segments and video key frame data in the archive, performing character recognition, image area detection and video frame screening and the like processing operations, further performing unified coding on the structure information and standardization processing on the spatial position information, and outputting the multi-modal archive content structure data with uniform format, to provide a basis for downstream feature processing.

[0262] The feature extraction module 20 is used for vectorizing modeling of information of different modalities in the content structure data, and extracting various semantic and visual features. Specifically, it includes: performing semantic coding processing on the text paragraphs based on a pre-trained NLP semantic model (such as BERT), to generate high-dimensional text paragraph feature vectors; performing regional feature extraction on the image segments based on a visual coding model (such as ResNet), to generate image segment feature vectors; and performing frame-level semantic modeling on the key frames based on a video understanding network, to obtain video frame feature representations. Finally, three types of modality feature vectors are output, to constitute a basic feature set for subsequent fusion modeling.

[0263] The fusion modeling module 30 is used for modality alignment and joint modeling of the extracted multi-modal feature vectors, to generate a fusion semantic representation structure that can be used for unified classification reasoning. First, modality alignment processing is performed, to map feature vectors of different sources to a common semantic space; then a cross-modal fusion modeling network is called, to fuse text, image and video information in a dynamic weighting manner, to calculate a fusion representation vector; finally, a context attention mechanism is introduced based on the fusion vector, to mine the context dependency between features, to form a fusion semantic representation structure. This module is the core modeling structure of the application, and directly determines the semantic integrity of subsequent label generation.

[0264] The label generation module 40 performs a hierarchical label generation operation based on the fused semantic representation structure, specifically including three stages of main label generation, sub-label generation and keyword extraction. The system first calls the main label generation model to predict the main semantic label set of the archive according to the fused representation; then, through the hierarchical decoder structure, the semantic path control mechanism is combined to further generate the sub-label set and the keyword set. The above label data is processed by hierarchical combination according to the semantic structure, and finally the multi-level label structure data is output, providing a basic structure for semantic classification and label indexing.

[0265] The label optimization module 50 performs semantic clustering and label reconstruction processing on the preliminary label structure data to improve the accuracy and clarity of the label set. Specifically, it includes: inputting the label clustering model to calculate the semantic distance and eliminate redundancy of the main label, sub-label and keyword; reconstructing the label system based on the clustering structure to form an optimized label set; constructing a label structure graph based on the optimization result to establish the semantic inheritance path and the correlation path between the labels, and providing graphical support for the structural rationality of label generation.

[0266] The feedback optimization module 60 realizes the comparative analysis and model feedback update between the label graph and the fused representation structure. The system first calculates the label coverage rate and consistency index, identifies the deviation area of the model generated label through path alignment and semantic difference analysis; based on the analysis result, automatically adjusts the attention parameters in the cross-modal fusion model and the label generation model to enhance the response ability of the model to high-value label semantics; finally, the updated parameters are re-injected into the fused semantic representation process to complete the closed-loop feedback optimization of the model end.

[0267] The user interface and model management module 70 provides graphical configuration interface, task management console and model parameter monitoring function to realize visual control and model running management of the system. Users can upload archive data, view label generation results, export multi-level label structure, and perform model tuning and training strategy adjustment operations to ensure the stability and adaptability of the system in multiple scene applications.

[0268] The beneficial effects of the embodiment are:

[0269] (1) Realize multi-modal fusion unified classification representation: the system realizes the semantic fusion of text, image and video information through modal alignment and dynamic attention mechanism, solves the low classification accuracy and semantic fragmentation problem caused by modal heterogeneity in traditional systems, and significantly improves the classification uniformity and semantic consistency of multi-source archives.

[0270] (2) Promote the structured degree and interpretability of label generation: The system introduces a dynamic tree decoding and label hierarchy combination mechanism to generate a multi-level label system containing main labels, sub-labels and keywords, avoiding the drawbacks of traditional label systems such as flatness and high redundancy, and enhancing the semantic expression ability and traceability of the label structure.

[0271] (3) Form a closed-loop optimized label feedback mechanism: Based on the bidirectional consistency analysis results of the label atlas and the fusion representation structure, the system automatically adjusts the attention parameters and modal weights of the model, realizes the adaptive evolution ability of the model level, and continuously improves the performance of the model in the multi-modal complex scene.

[0272] (4) Support the integrated deployment of intelligent classification and label recommendation: The seven modules constructed by the present application can be deployed in the archive management platform, support automatic classification, semantic label generation and retrieval enhancement services for large-scale archives, and have wide industrial application potential and social value.

[0273] Obviously, the above-described embodiments are only part of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and on the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by using the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for intelligent classification and tag generation of multimodal archives, characterized in that, include: The system acquires text paragraphs, image fragments, and video keyframe data from the files to be processed, and generates multimodal file content structure data through format recognition, content region separation, and standardization processing. Based on multimodal archive content structure data, semantic encoding, image region feature encoding, and frame-level feature modeling are performed to extract feature vectors from text paragraphs, image segments, and video keyframes. Modal alignment, cross-modal joint modeling, and contextual attention extraction are performed on the feature vectors of text paragraphs, image segments, and video keyframes to generate a fused semantic representation structure; Based on the fusion semantic representation structure, a set of main tags, a set of sub-tags, and a set of keywords are generated. Hierarchical combination processing is then performed to construct multi-level tag structure data. The label structure is implemented by constructing a graph structure, and a multi-level label graph is defined as follows: Among them, V (n) E is a set of tag nodes. (n) For the set of semantic edges of the label; Ranking labels by importance using the PageRank algorithm: in, V represents the weight of label node i in the t-th iteration; (n) Let δ be the set of nodes; δ is the jump probability. Let i be the set of adjacent nodes of label i; The expression for the multi-level tag structure is: in, The label graph structure for the nth sample; V (n) For a set of nodes; E (n) Let π be the set of edges; (n) Here, μ is the node weight vector; μ is the Möbius function level mapping strength. Perform semantic redundancy removal and label reconstruction on multi-level label structure data to generate an optimized label set and label structure graph; Consistency analysis and update processing are performed based on the tag structure graph and the fusion semantic representation structure to complete the feedback optimization of the cross-modal fusion model and the tag generation model.

2. The multimodal archive intelligent classification and tag generation method according to claim 1, characterized in that, The steps for generating multimodal archive content structure data include: Preliminary parsing results are obtained through format recognition and content region separation operations; Perform character recognition, target region detection, and frame filtering to obtain a content structure set; The content structure set is encoded uniformly and its location information is standardized to generate multimodal archive content structure data.

3. The multimodal archive intelligent classification and tag generation method according to claim 1, characterized in that, The steps for obtaining text paragraphs, image fragments, and video keyframe feature vectors also include: Obtain text paragraphs from multimodal archive content structure data, input them into a text semantic model for semantic encoding processing, and generate text paragraph feature vectors; Image fragments are obtained from the multimodal archive content structure data, input into the image semantic model for image region feature encoding, and image fragment feature vectors are generated. The video keyframes in the multimodal archive content structure data are obtained, input into the video semantic model for frame-level feature modeling, and the video keyframe feature vectors are generated.

4. The multimodal archive intelligent classification and tag generation method according to claim 1, characterized in that, Modal alignment processing is performed on the feature vectors of text paragraphs, image segments, and video keyframes to generate an initial alignment vector set. A cross-modal fusion model is constructed based on the initial alignment vector set, cross-modal joint modeling processing is performed, and context attention extraction processing is performed on the fused semantic representation vector to generate a fused semantic representation structure.

5. The multimodal archive intelligent classification and tag generation method according to claim 1, characterized in that, Obtain the fused semantic representation structure, input it into the label generation model to perform the main label generation operation, and generate a preliminary set of main labels.

6. The multimodal archive intelligent classification and tag generation method according to claim 5, characterized in that, Based on the initial set of main tags and the fused semantic representation structure, tag-level decoding is performed to generate a set of sub-tags and keywords.

7. The multimodal archive intelligent classification and tag generation method according to claim 6, characterized in that, The main tag set, sub tag set, and keyword set are combined hierarchically to generate multi-level tag structure data.

8. The multimodal archive intelligent classification and tag generation method according to claim 1, characterized in that, The steps of generating the optimized tag set and tag structure map also include: Obtain multi-level label structure data, input it into the label clustering model for semantic redundancy removal, and obtain the label clustering structure; The tag clustering structure is reconstructed to generate an optimized tag set with a clear semantic structure; Generate a tag structure graph based on an optimized tag set, and construct semantic association paths between tags.

9. The multimodal archive intelligent classification and tag generation method according to claim 1, characterized in that, The steps for completing the feedback optimization of the cross-modal fusion model and the label generation model also include: Obtain the label structure graph and the fused semantic representation structure, and perform label coverage and consistency analysis. Based on the analysis results, update the attention parameters in the cross-modal fusion model and the label generation model; The updated attention parameters are applied to the fusion semantic representation structure generation process to complete the model feedback optimization and update.

10. A multimodal archive intelligent classification and tag generation system, applied to the method of any one of claims 1-9, characterized in that, include: The content structure extraction module is used to acquire text paragraphs, image fragments, and video keyframe data to generate multimodal archive content structure data; The feature extraction module is used to extract text paragraph feature vectors, image segment feature vectors, and video keyframe feature vectors from multimodal archive content structure data. The fusion modeling module is used for modality alignment, cross-modal joint modeling, and contextual attention extraction to generate a fusion semantic representation structure. The tag generation module is used to generate multi-level tag structure data based on the fused semantic representation structure. The tag optimization module is used to perform semantic redundancy removal and tag reconstruction on multi-level tag structure data to generate a tag structure graph. The feedback optimization module is used to update the attention parameters of the cross-modal fusion model and the label generation model based on the label structure graph and the fusion semantic representation structure. The user interface and model management module provide a graphical configuration interface, a task management console, and model parameter monitoring functions. Users can upload file data, view tag generation results, and export multi-level tag structures through the interface.

Images