Active masking and dynamic learning method, system and training tool based on multi-modal semantic unit

Abstract

The invention relates to an active shielding and dynamic learning method, system and training tool based on a multi-modal semantic unit, and provides an active shielding mechanism based on semantic unit importance and model state self-adaption. According to the mechanism, firstly, through cross-modal attention calculation, a key semantic unit with a high correlation degree between vision and a text is screened out, and then a shielding target is dynamically determined in combination with the current prediction difficulty of a model, so that the training process is more focused on a high-value region which is not fully learned by the model. Through the closed-loop optimization design of cross-modal association, difficulty assessment and active shielding, the learning efficiency and generalization ability of the model can be significantly improved. According to the method, on one hand, the problem of low information utilization rate existing in traditional random masks is effectively solved; and on the other hand, accurate alignment and efficient learning of the visual salient region and the text key information are realized, so that the model shows higher robustness and adaptivity in the aspects of semantic understanding, visual association, knowledge migration and the like.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, specifically to an active masking and dynamic learning method and system based on multimodal semantic units. Background Technology

[0002] With the development of artificial intelligence, multimodal learning has become an important research direction in perceptual and cognitive intelligence. Its core goal is to enable models to understand and associate information from different modalities (such as images, text, speech, etc.), thereby achieving cross-modal alignment, semantic understanding, and knowledge reasoning.

[0003] To achieve high-quality multimodal representations, masked pre-training has become one of the mainstream methods. This method randomly masks a portion of the input (such as a text token or image patch), allowing the model to predict the masked portion based on the context, thereby learning the underlying semantic representation. This idea originated from the BERT model in the field of natural language processing and has been extended to the multimodal domain through works such as VisualBERT and UNITER. These works jointly input image region features and text sequences into a Transformer structure and use a joint masking strategy for pre-training.

[0004] However, existing methods generally employ random masking or uniform sampling strategies, failing to consider the importance and learning difficulty of different semantic segments during training. This leads to two prominent problems: First, the model may waste computational resources on a large number of irrelevant or easily predictable segments, resulting in semantic redundancy and inefficient training. Second, random masking ignores the semantic correspondence between text and vision, making it difficult for the model to focus on key information with strong cross-modal correlation, thus limiting its generalization ability on complex semantic understanding tasks.

[0005] Further analysis reveals that existing technical solutions have the following three main limitations: Masking strategies lack initiative and adaptability: methods such as UNITER and ViLT use static random masks, which cannot dynamically adjust the mask target according to the model's current learning state or training feedback.

[0006] Semantic modeling granularity is inflexible: most methods use a single token as the mask unit, failing to fully consider the more complete semantic unit structure in natural language, such as phrase level and concept level.

[0007] Insufficient cross-modal consistency guarantees: Although models such as ALBEF and BLIP enhance interaction through cross-modal attention mechanisms, their masking process does not explicitly incorporate visual saliency constraints, leading to deviations in cross-modal alignment. While recent entity-level masking methods (such as METER) have shown improvement, their selection mechanisms remain static or based on fixed rules, lacking dynamic difficulty assessment based on model prediction uncertainty. Furthermore, existing pre-training methods are mostly embedded as algorithm modules in the overall process, lacking dedicated training tools for researchers or developers, making it difficult to support flexible tuning of training strategies and interactive experimentation.

[0008] Therefore, existing multimodal pre-training methods are insufficient in terms of the initiative of masking strategies, the flexibility of semantic granularity, and the accuracy of cross-modal alignment. There is an urgent need for an innovative solution that can achieve adaptive and efficient learning. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multimodal joint learning method based on active masking semantic units. Its innovation lies in introducing "semantic units" to replace the traditional concept of "entities," constructing a semantic hierarchy through text segmentation and semantic recognition; and dynamically selecting the most valuable semantic units for masking prediction in each training round through an active masking mechanism and a difficulty-aware strategy. This method integrates visual saliency, cross-modal attention, and model prediction uncertainty, forming an adaptive semantic selection, masking, and learning loop mechanism, significantly improving the model's learning efficiency and cross-modal understanding ability. Simultaneously, this invention provides a visual training tool, encapsulating the above process into an interactive and monitorable system, improving the ease of use of the method.

[0010] In a first aspect, the present invention provides an active masking and dynamic learning method based on multimodal semantic units, comprising the following steps: Multimodal input representation steps: acquire images and corresponding text descriptions to form a multimodal sample set; perform semantic recognition and segmentation on the text descriptions to obtain multiple semantic units; and use a vision-language joint encoding architecture to represent multimodal features. Cross-modal association modeling and key semantic unit screening steps: Based on the cross-modal attention mechanism, calculate the semantic attention intensity between the visual region of the image and the semantic unit, and screen out the key semantic unit candidate set from all semantic units according to the semantic attention intensity; The steps for semantic unit difficulty assessment based on model state are as follows: mask the semantic units in the candidate set of key semantic units, make predictions using the current model parameters, calculate the prediction loss of each masked semantic unit, and use the prediction loss as the difficulty assessment index of the semantic unit. Active occlusion selection and model training steps: Based on the difficulty evaluation index, select one or more semantic units with the highest difficulty from the candidate set of key semantic units to form the final occlusion set for occlusion, and use this to build training samples to train the model and update parameters.

[0011] Based on the above scheme, the semantic unit is a language segment consisting of one or more sub-words that carries independent semantic information.

[0012] Based on the above scheme, the specific implementation of the vision-language joint encoding architecture for representing multimodal features includes: Extract visual vectors from image features; Extract semantic representation vectors from the text description; By modeling the interaction features between visual vectors and semantic representation vectors using an attention mechanism, a fused vector is obtained.

[0013] Based on the above scheme, the key semantic unit filtering step specifically includes: Calculate the attention matrix between the image region features and text word features of the fused vector to obtain the semantic attention strength between the image region and the text word; The average attention scores of all words belonging to the same semantic unit are calculated to obtain the overall semantic attention score of that semantic unit. Based on the overall semantic relevance score, the semantic units are sorted, and a predetermined number of the top-ranked semantic units are selected to form the candidate set of key semantic units.

[0014] Based on the above scheme, when screening the candidate set of key semantic units, a visual saliency constraint is further introduced, which only allows semantic units with high alignment in visually salient regions.

[0015] Based on the above scheme, in the semantic unit difficulty assessment step, the prediction loss is the cross-entropy loss.

[0016] Based on the above scheme, after the active occlusion and model training steps, an adaptive iteration step is also included: After the model parameters are updated, the key semantic unit selection step, the semantic unit difficulty evaluation step, and the active occlusion and mutual information optimization training step are repeated to form a closed-loop active learning cycle until the model converges on the validation set or reaches the predetermined number of training rounds.

[0017] Based on the above scheme, in the active occlusion selection and model training steps, the model is optimized by maximizing the mutual information between the occluded semantic units and the corresponding visual regions to enhance cross-modal semantic alignment.

[0018] Secondly, a multimodal semantic unit-based active masking and dynamic learning system for performing the method is provided, comprising: Multimodal input representation module: configured to acquire images and corresponding text descriptions to form a multimodal sample set; perform semantic recognition and segmentation on the text descriptions to obtain multiple semantic units; and use a vision-language joint encoding architecture to represent multimodal features; Cross-modal association modeling and key semantic unit screening module: configured to calculate the semantic attention intensity between the visual region of the image and the semantic unit based on the cross-modal attention mechanism, and screen out a candidate set of key semantic units from all semantic units based on the semantic attention intensity; The semantic unit difficulty assessment module based on model state is configured to mask semantic units in the candidate set of key semantic units, make predictions using the current model parameters, calculate the prediction loss of each masked semantic unit, and use the prediction loss as the difficulty assessment index of the semantic unit. Active occlusion selection and mutual information optimization training module: configured to select one or more semantic units with the highest difficulty from the candidate set of key semantic units according to the difficulty evaluation index to form a final occlusion set for occlusion, and use this to construct training samples to train the model and update parameters.

[0019] Thirdly, the present invention provides a visualization training tool for implementing the above-described method, characterized in that it comprises: Strategy configuration module: used for interactive setting of semantic unit segmentation rules, occlusion ratio, attention mechanism parameters, difficulty evaluation indicators and visual saliency constraints; Training process monitoring module: used to display the occluded semantic units, model loss change curves, and cross-modal attention distribution in real time; Interactive debugging module: Allows users to manually adjust the occlusion units and observe the model's prediction feedback, providing real-time performance evaluation during the training process.

[0020] Fourthly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0021] Fifthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.

[0022] The beneficial effects of this invention are: This invention introduces a proactive and adaptive masking and learning mechanism by employing semantic unit-level modeling, cross-modal attention-driven key unit selection, and dynamic difficulty assessment based on model prediction loss. This method enables the model to actively select semantic content with both cross-modal relevance and high learning difficulty for learning during training, significantly avoiding the blindness of random masking and thus effectively improving the efficiency of multimodal pre-training and the model's final semantic understanding ability. Pre-training on the CC3M and COCO datasets and testing on downstream task datasets such as VQA-V2 and SNLI-VE both improved the average prediction accuracy by more than one percentage point. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating an embodiment of the active occlusion and dynamic learning method provided by the present invention. Detailed Implementation

[0024] To make the objectives, advantages and features of the present invention more apparent, the present invention will be further described in detail below with reference to specific embodiments.

[0025] This embodiment provides an active masking and dynamic learning method based on multimodal semantic units, the process of which is as follows: Figure 1 As shown, the specific steps include: S1: Multimodal input representation.

[0026] S11: Collect a multimodal sample set containing an image I and its corresponding text description T.

[0027] This sample set can be represented as:

[0028] in, Indicates the first A visual input, Indicates the related first There are 3 text description sequences, where N is the total number of samples.

[0029] S12: Text description Perform semantic recognition and segmentation to divide the data into multiple sets of semantic units. Each semantic unit A token is a language segment consisting of several tokens that carries independent semantic information, such as a noun phrase or a verb phrase. m is the number of semantic units in the sample.

[0030] S13: Feature representation is performed using a vision-language joint encoding architecture. The image feature extraction module outputs a visual vector: ; The text encoding module outputs a semantic representation vector: ; The cross-modal fusion module models the interaction features of the two modalities through an attention mechanism to obtain a fusion vector: ; After this stage, the system can obtain each semantic unit and its cross-modal fusion representation, laying the foundation for subsequent active masking strategies.

[0031] S2: Cross-modal association modeling and key semantic unit selection.

[0032] S21: Cross-modal attention computation.

[0033] Based on the fused feature representation h obtained in step S1, the cross-modal attention matrix A is calculated, and its elements are... This reflects the semantic attention intensity between the p-th image region and the q-th text token.

[0034] S22: Filtering of key semantic units.

[0035] For each semantic unit The overall semantic attention score of the unit is obtained by averaging the attention scores of all J subwords within it. Score , where l ranges from 1 to J, and the attention values ​​corresponding to all sub-words l belonging to this unit are summed.

[0036] All semantic units are ranked based on this score, and the top-ranked units are selected. The candidate set U_C is composed of the units with the highest scores.

[0037] in: This represents the number of semantic units that need to be masked in each round of training. A multiplier hyperparameter greater than 1 is used to increase the diversity of the candidate set.

[0038] Optionally, visual saliency constraints can be introduced in this process to retain only those semantic units that are strongly aligned with regions of high visual saliency response in the image, so as to ensure that the candidate set has high consistency across modalities.

[0039] S3: Semantic unit difficulty assessment based on model state.

[0040] To avoid the model repeatedly learning easily predictable semantic segments, an active learning mechanism is introduced to dynamically evaluate and rank candidate semantic units based on their difficulty.

[0041] Each semantic unit in the candidate set U_C Perform occlusion sequentially (replace with [MASK]), utilizing the current model parameters. Perform forward prediction to obtain the predicted probability distribution of the occluded cells. , where s is the obscured content and c is the context.

[0042] Then, the prediction loss for this unit is calculated, for example, using cross-entropy loss: ,right Sum the losses of all subwords s in the expression.

[0043] This loss value This serves as a difficulty assessment metric for measuring the uncertainty of the current model's understanding of the semantic unit.

[0044] S4: Active occlusion selection and mutual information optimization training.

[0045] S41: Active occlusion selection.

[0046] Based on the difficulty assessment index (i.e., predicted loss) calculated in step S3 From the candidate set U_C, the k semantic units with the highest prediction loss are selected to form the final masking set U_M used for training. This strategy ensures that the masked content has both cross-modal connections and high learning value.

[0047] S42: Mutual information optimization training.

[0048] Optimize model parameters through backpropagation The goal is to minimize the model's prediction loss when recovering occluded semantic units. Simultaneously, by introducing loss terms such as cross-modal contrastive learning, the aim is to maximize the mutual information between occluded semantic units and their corresponding visual regions, thereby enhancing cross-modal semantic alignment.

[0049] S5: Adaptive Iteration.

[0050] After completing one round of training and parameter updates, the model parameters are updated to: The system automatically jumps back to step S2, recalculates the cross-modal attention and semantic unit difficulty based on the new model state, and begins a new round of "selection-evaluation-training" cycle. This process continues, forming a dynamic and adaptive active learning loop, until the model's performance on the independent validation set converges or the preset maximum number of training rounds is reached.

[0051] Based on the same inventive concept, another embodiment of the present invention also provides a system corresponding to the above method, comprising: Multimodal input representation module: configured to acquire images and corresponding text descriptions to form a multimodal sample set; perform semantic recognition and segmentation on the text descriptions to obtain multiple semantic units; and use a vision-language joint encoding architecture to represent multimodal features; Cross-modal association modeling and key semantic unit screening module: configured to calculate the semantic attention intensity between the visual region of the image and the semantic unit based on the cross-modal attention mechanism, and screen out a candidate set of key semantic units from all semantic units based on the semantic attention intensity; The semantic unit difficulty assessment module based on model state is configured to mask semantic units in the candidate set of key semantic units, make predictions using the current model parameters, calculate the prediction loss of each masked semantic unit, and use the prediction loss as the difficulty assessment index of the semantic unit. Active occlusion selection and mutual information optimization training module: configured to select one or more semantic units with the highest difficulty from the candidate set of key semantic units according to the difficulty evaluation index to form a final occlusion set for occlusion, and use this to construct training samples to train the model and update parameters.

[0052] Based on the same inventive concept, another embodiment of the present invention also provides a visualization training tool, which encapsulates the system into a software application with an interactive interface, specifically including: Strategy configuration module: Provides a graphical parameter setting panel for configuring semantic segmentation models, key unit selection thresholds, training epochs, and optimizer hyperparameters; Training process monitoring module: Dynamically visualizes the selection status of key semantic units, the trend of loss function changes, and the image-text attention association heatmap during the training process; Interactive debugging module: Allows users to pause and manually intervene in occlusion selection during training, perform single-step training, and perform real-time performance verification based on pre-loaded validation sample sets.

[0053] It should be noted that any process or method description in the embodiments can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which the embodiments of the invention pertain.

[0054] It should be noted that the logic and / or steps in the embodiments, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0055] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0056] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0057] Furthermore, in the embodiments of the present invention, the functional modules can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0058] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0059] The above embodiments have provided a detailed description of the technical solutions of the present invention. Obviously, the present invention is not limited to the described embodiments. Based on the embodiments of the present invention, those skilled in the art can make various changes, but any changes that are equivalent or similar to the present invention fall within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. An active masking and dynamic learning method based on multimodal semantic units, characterized in that, Includes the following steps: Multimodal input representation steps: acquire images and corresponding text descriptions to form a multimodal sample set; perform semantic recognition and segmentation on the text descriptions to obtain multiple semantic units; and use a vision-language joint encoding architecture to represent multimodal features. Cross-modal association modeling and key semantic unit screening steps: Based on the cross-modal attention mechanism, calculate the semantic attention intensity between the visual region of the image and the semantic unit, and screen out the candidate set of key semantic units from all semantic units according to the semantic attention intensity; The steps for semantic unit difficulty assessment based on model state are as follows: mask the semantic units in the candidate set of key semantic units, make predictions using the current model parameters, calculate the prediction loss of each masked semantic unit, and use the prediction loss as the difficulty assessment index of the semantic unit. Active occlusion selection and model training steps: Based on the difficulty evaluation index, select one or more semantic units with the highest difficulty from the candidate set of key semantic units to form the final occlusion set for occlusion, and use them to build training samples to train the model and update parameters. Optimize the model by maximizing the mutual information between the occluded semantic units and the corresponding visual regions to enhance cross-modal semantic alignment.

2. The method according to claim 1, characterized in that, The semantic unit is a language segment consisting of one or more sub-words, carrying independent semantic information.

3. The method according to claim 2, characterized in that, The specific implementation of the vision-language joint coding architecture for representing multimodal features includes: Extract visual vectors from image features; Extract semantic representation vectors from the text description; By modeling the interaction features between visual vectors and semantic representation vectors using an attention mechanism, a fused vector is obtained.

4. The method according to claim 3, characterized in that, The key semantic unit filtering step specifically includes: Calculate the attention matrix between the image region features and text word features of the fused vector to obtain the semantic attention strength between the image region and the text word; The average attention scores of all words belonging to the same semantic unit are calculated to obtain the overall semantic attention score of that semantic unit. Based on the overall semantic relevance score, the semantic units are sorted, and a predetermined number of the top-ranked semantic units are selected to form the candidate set of key semantic units.

5. The method according to claim 4, characterized in that, When screening the candidate set of key semantic units, a visual saliency constraint is further introduced, which only includes semantic units with high alignment in visually salient regions.

6. The method according to claim 1, characterized in that, In the semantic unit difficulty assessment step, the prediction loss is the cross-entropy loss.

7. The method according to claim 1, characterized in that, Following the active masking and mutual information optimization training steps, an adaptive iterative step is also included: After the model parameters are updated, the key semantic unit selection step, the semantic unit difficulty evaluation step, and the active occlusion and mutual information optimization training step are repeated to form a closed-loop active learning cycle until the model converges on the validation set or reaches the predetermined number of training rounds.

8. An active masking and dynamic learning system based on multimodal semantic units for performing the method according to any one of claims 1 to 7, characterized in that, include: Multimodal input representation module: configured to acquire images and corresponding text descriptions to form a multimodal sample set; perform semantic recognition and segmentation on the text descriptions to obtain multiple semantic units; and use a vision-language joint encoding architecture to represent multimodal features; Cross-modal association modeling and key semantic unit screening module: configured to calculate the semantic attention intensity between the visual region of the image and the semantic unit based on the cross-modal attention mechanism, and screen out a candidate set of key semantic units from all semantic units based on the semantic attention intensity; The semantic unit difficulty assessment module based on model state is configured to mask semantic units in the candidate set of key semantic units, make predictions using the current model parameters, calculate the prediction loss of each masked semantic unit, and use the prediction loss as the difficulty assessment index of the semantic unit. Active occlusion selection and mutual information optimization training module: configured to select one or more semantic units with the highest difficulty from the candidate set of key semantic units according to the difficulty evaluation index to form a final occlusion set for occlusion, and use this to build training samples to train the model and update parameters, and optimize the model by maximizing the mutual information between the occluded semantic units and the corresponding visual regions.

9. A visualization training tool for performing the method according to any one of claims 1 to 7, characterized in that, include: The strategy configuration module is used to interactively set semantic unit segmentation, masking strategies, and model training parameters; The training process monitoring module is used to display the selection of occlusion units and changes in training metrics in real time; An interactive debugging module is used to support manual occlusion intervention and step-by-step execution of the training process.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images