Function word extraction system and method based on multi-modal adaptive learning

Abstract

The invention relates to an efficacy word extraction system and method based on multi-modal adaptive learning. According to the system, multi-modal data such as texts, images and audios are received through the input module, and feature fusion is carried out through the preprocessing module. A hybrid encoder module is utilized, local n-gram features are firstly extracted through a convolutional neural network layer, then global context information is extracted through a Transform encoder, the comprehensiveness of feature representation is guaranteed through a cooperation mechanism, and the limitation of a traditional single model in the aspect of feature representation precision is overcome. And then, the knowledge distillation module optimizes the text semantic representation data to generate efficient lightweight semantic representation information. The decoder module integrates a vocabulary library, adopts a strategy of combining generation and copying, and utilizes the thought of a pointer-generation network, so that the problem of unregistered words is effectively solved, the coverage rate and accuracy are ensured while generalization is kept, and efficient, accurate and robust efficacy word extraction is realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a system and method for extracting efficacy words based on multimodal adaptive learning. Background Technology

[0002] With the continuous development of computer technology, the field of natural language processing has made significant progress. Efficacy keyword extraction technology plays a crucial role in numerous scenarios such as literature analysis, product recommendation, and social media monitoring. Accurate extraction of efficacy keywords helps businesses understand market demands, grasp product characteristics, and provide strong support for decision-making. It also enables timely identification of user concerns and feedback in social media monitoring, improving user experience. However, efficacy keyword extraction faces many challenges, requiring continuous technological improvements to adapt to complex and ever-changing application scenarios.

[0003] In related technologies, invention patent application publication number CN120821890A discloses a method, device, and medium for identifying sensitive words and intentions based on multimodality. This method extracts features from data of different modalities, preserving specific information for each modality; it also directly extracts features from multimodal features, capturing potential associations and interactions between different modalities. Thus, in the feature extraction stage, the original data or feature vectors of different modalities are effectively fused, fully exploring the associations and complementarities between modal information, achieving comprehensive screening of multimodal information and avoiding information omissions caused by modality separation. Simultaneously, through multi-dimensional methods such as sentiment analysis, semantic logic analysis, and behavioral pattern analysis, the intent of information is deeply mined, thereby not only identifying sensitive words but also understanding the true intent of the publisher, enabling more targeted measures to maintain a healthy and secure online environment.

[0004] Chinese patent application publication number CN118797038A discloses a multimodal text summarization method, which specifically includes: acquiring image data and text data, and preprocessing them separately; extracting visual features of the image based on a convolutional neural network; extracting features from the text data using natural language processing techniques, obtaining semantic features of the text through a bag-of-words model and TF-IDF, wherein the semantic features include term frequency and inverse document frequency; fusing the features of the image and text to obtain a comprehensive multimodal feature vector; the fusion method includes simple concatenation, weighted fusion, and MLP modal fusion; constructing a text generation model to generate summary text by taking the multimodal feature vector as input; evaluating the generated summary text using the ROUGE evaluation metric; and optimizing the model based on the evaluation results.

[0005] However, the aforementioned technologies still have certain limitations. For example, although patent application CN120821890A achieves multimodal feature fusion and local feature extraction, it relies solely on convolutional neural networks and lacks effective capture of global contextual information. This results in insufficient feature representation accuracy when handling long-distance dependencies, affecting the recognition accuracy of efficacy words in complex semantic scenarios. As another example, although patent application CN118797038A uses a Transformer network to process multimodal features, its model structure is typically large, with low inference efficiency, making it difficult to deploy in resource-constrained environments (such as mobile devices or edge computing devices). Furthermore, when performing text generation or information extraction, most of the aforementioned technologies rely on fixed vocabularies. When faced with out-of-vocabulary (OOV) words (such as newly emerging efficacy ingredient names, specific product terms, or online slang), they often cannot be accurately generated or extracted, resulting in limited coverage and accuracy of efficacy words. Summary of the Invention

[0006] To improve the inference efficiency of the model and effectively solve the out-of-vocabulary (OV) word problem while ensuring comprehensive representation of multimodal features, this application provides a system and method for extracting efficacy words based on multimodal adaptive learning.

[0007] On the one hand, this application provides a efficacy word extraction system based on multimodal adaptive learning, which adopts the following technical solution: A efficacy word extraction system based on multimodal adaptive learning includes: The input module is used to receive raw data and extract multimodal data from the raw data, wherein the multimodal data includes text information, image data and audio data; A preprocessing module, connected to the input module, is used to perform feature fusion on the multimodal data to generate a fused feature vector; A hybrid encoder module is connected to the preprocessing module. The hybrid encoder module includes a convolutional neural network layer and a Transformer encoder. The convolutional neural network layer is used to extract local n-gram feature information from the fused feature vector, and the Transformer encoder is used to extract global context information from the fused feature vector and the local n-gram feature information to generate text semantic representation data. A knowledge distillation module, connected to the hybrid encoder module, is used to distill and optimize the text semantic representation data to generate lightweight semantic representation information; A decoder module, connected to the knowledge distillation module, integrates a vocabulary library. The decoder module calculates the generation probability and copy probability based on the lightweight semantic representation information, and dynamically switches between generation and copy modes to generate efficacy words based on the generation and copy probabilities. If the generation mode is selected, the efficacy words are selected from preset words in the vocabulary library; if the copy mode is selected, the efficacy words are copied from the lightweight semantic representation information via a pointer mechanism.

[0008] The above technical solution receives multimodal data, including text, images, and audio, through an input module, and performs feature fusion through a preprocessing module. A hybrid encoder module first extracts local n-gram features through a convolutional neural network layer, and then extracts global contextual information through a Transformer encoder. This collaborative mechanism ensures the comprehensiveness of feature representation and overcomes the limitations of traditional single models in feature representation accuracy. Subsequently, a knowledge distillation module optimizes the text semantic representation data, generating efficient and lightweight semantic representation information. Finally, the decoder module integrates a vocabulary and adopts a strategy combining generation (matching preset words) and copying (not matching preset words). This essentially utilizes the idea of ​​a pointer-generation network, effectively solving the out-of-vocabulary (OOV) word problem, ensuring coverage and accuracy while maintaining generalization, thus achieving efficient, accurate, and robust functional word extraction. Comparative experiments verify that the F1 score of the proposed model (90.3%) is significantly higher than that of the baseline model (79.5%) and the contrasting model (86.1%). When handling out-of-vocabulary (OOV) words, the F1 score of our model (78.6%) is approximately 149% higher than the comparison model (31.5%). Furthermore, our model is optimized through a knowledge distillation module, reducing the number of parameters to approximately 42M, a reduction of approximately 61.8%. Simultaneously, the inference latency is reduced from 48ms to 21ms, resulting in an inference speed improvement of approximately 128%.

[0009] Optionally, the preprocessing module is used to perform feature fusion on the multimodal data to generate a fused feature vector; wherein, the preprocessing module is configured to: perform semantic-level filtering and standardization on the text information to generate text feature information; convert the text feature information into a text feature vector; extract visual feature information from the image data; convert the visual feature information into a visual feature vector; perform Mel-frequency cepstral coefficient processing on the audio data and extract auditory feature information; convert the auditory feature information into an auditory feature vector; and perform feature fusion on the text feature vector, the visual feature vector, and the auditory feature vector to generate the fused feature vector.

[0010] By employing the above technical solution, the preprocessing module performs semantic-level filtering and standardization on text information, which helps to identify and remove redundant or low-information sentences. Simultaneously, visual features are extracted from image data, and auditory features are extracted from audio data through Mel-frequency cepstral coefficient processing. These feature vectors are then converted to a format compatible with text feature vectors, and finally, the text, visual, and auditory feature vectors are fused. This approach ensures effective processing and adaptation of data from different modalities, achieving comprehensive utilization of multimodal information and providing high-quality fused feature vectors for the subsequent hybrid encoder module.

[0011] Optionally, the convolutional neural network layer is used to extract local n-gram feature information from the fused feature vector; wherein, the convolutional neural network layer is configured as follows: Using multiple convolution kernels of different sizes W (k) For fused feature vectors Convolution processing is performed, expressed as follows: in, Let L represent the matrix dimension of the fused feature vector E, L represent the sequence length, and D represent the feature dimension. W represents the local feature information extracted by a convolutional kernel of size k×D. (k) This represents a convolution kernel of size k×D, b (k) The bias term, E, represents the convolution kernel of size k×D. i:i+k-1 ReLU(.) represents a local k-gram segment of length k starting from position i in the fused feature vector E, and ReLU(.) represents the activation function. For multiple convolution kernels W of different sizes (k) The extracted local feature information is integrated to generate local n-gram feature information F. CNN It can be expressed as the following formula: F CNN =Concat(Pooling(F (3) Pooling(F) (5) ),……) Here, Concat(.) represents concatenation operation and Pooling(.) represents pooling operation.

[0012] The Transformer encoder is used to extract global context information from the fused feature vector and the local n-gram feature information to generate text semantic representation data; wherein, the Transformer encoder is configured to: extract global context information from the fused feature vector E and the local n-gram feature information F CNNFeature integration is performed to generate the initial input vector H of the Transformer encoder. (0) The initial input vector H is calculated using a multi-head self-attention mechanism. (0) The correlation between any two positions in the vector is determined to extract global context information; the output vector H after multi-layer Transformer encoding is then processed. (Final) R as text semantic representation data Semantic .

[0013] By employing the aforementioned technical solution, the convolutional neural network layer uses multiple convolutional kernels of different sizes to convolve the fused feature vector and integrates the extracted local feature information, thereby capturing local n-gram feature information of different granularities. The Transformer encoder then integrates the fused feature vector with this local n-gram feature information and utilizes a multi-head self-attention mechanism to calculate the correlation between any two positions, thus capturing long-range dependencies and global contextual information in the sequence. This collaborative approach, where CNN extracts local features and Transformer captures global dependencies, provides structured local information for the global dependency capture unit, avoiding information loss and ensuring the comprehensiveness of feature representation, thus solving the problem of insufficient feature representation accuracy in traditional single models.

[0014] Optionally, the knowledge distillation module includes a teacher model unit, a student model unit, and a distillation learning and optimization unit; the teacher model unit is used to perform data reasoning based on the text semantic representation data to generate soft target data; the student model unit is used to perform imitation reasoning based on the text semantic representation data to generate student imitation data; the distillation learning and optimization unit is used to calculate the difference between the student imitation data and the soft target data to determine the distillation loss, and to minimize the weighted sum of the distillation loss and the hard target loss through iterative training to generate the lightweight semantic representation information.

[0015] Using the above technical solution, the knowledge distillation module uses a high-performance, large teacher model unit to infer textual semantic representation data, generating knowledge-rich soft-objective data. Simultaneously, a compact and computationally efficient lightweight student model unit performs imitation inference on the same textual semantic representation data, generating student imitation data. The distillation learning and optimization unit determines the distillation loss by calculating the difference between the two and minimizes the weighted sum of the distillation loss and the hard-objective loss. This approach successfully transfers the "knowledge" of the large teacher model to the lightweight student model, thereby significantly improving the model's inference speed and deployment efficiency while maintaining high performance as much as possible, ultimately generating efficient lightweight semantic representation information.

[0016] Optionally, the efficacy word extraction system based on multimodal adaptive learning further includes: An incremental learning module, connected to the hybrid encoder module, the knowledge distillation module, and the decoder module, is used to dynamically update the model parameters of the hybrid encoder module, the knowledge distillation module, and the decoder module through incremental learning.

[0017] By adopting the above technical solution and adding an incremental learning module, the system can collect new data, user feedback, and error cases encountered in the production environment. This data is then used to dynamically update the model parameters of the hybrid encoder, knowledge distillation, and decoder modules through incremental learning. This enables the model to adapt to dynamic text streams, minimizing the forgetting of old knowledge while learning new knowledge, thus ensuring the model's continuous adaptive capability to new data.

[0018] Optionally, the efficacy word extraction system based on multimodal adaptive learning further includes: An interpretability module, connected to the decoder and the hybrid encoder, is used to generate a visual interface to provide explanatory information to the user. The interpretability module is configured to: receive contextual attention weight information corresponding to the efficacy words when the decoder generates or replicates them; receive self-attention weight information corresponding to the text semantic representation data when the hybrid encoder generates text semantic representation data; generate an interpretability data package based on the efficacy words, the contextual attention weight information, the text semantic representation data, and the self-attention weight information; and generate a visual interface based on the interpretability data package.

[0019] By employing the above technical solution and adding an interpretability module, the system can receive contextual attention weight information during decoder operation and self-attention weight information during hybrid encoder operation. The interpretability module generates interpretive data packages based on this information and ultimately produces a visual interface. This transforms complex deep learning reasoning processes (such as the generation or replication logic of efficacy words) into user-understandable visual explanations (e.g., attention heatmaps or decision path diagrams), intuitively demonstrating the contribution of extraction decisions, thereby enhancing the system's traceability and user trust in the extraction results.

[0020] Another invention disclosed in this application is a method for extracting efficacy words based on multimodal adaptive learning, comprising the following steps: S10. Extract multimodal data from the raw data, wherein the multimodal data includes text information, image data and audio data; S20. Perform feature fusion on the multimodal data to generate a fused feature vector; S30. Extract local n-gram feature information from the fused feature vector through a convolutional neural network layer; S40. Extract global context information from the fused feature vector and the local n-gram feature information using a Transformer encoder to generate text semantic representation data; S50. Distillation optimization is performed on the text semantic representation data to generate lightweight semantic representation information; S60. Calculate the generation probability and the replication probability based on the lightweight semantic representation information; S70. Dynamically switch between generation mode and copying mode based on the generation probability and the copying probability to generate efficacy words; wherein, if the generation mode is selected, the efficacy words are selected from preset words in the vocabulary; if the copying mode is selected, the efficacy words are copied from the lightweight semantic representation information through a pointer mechanism.

[0021] The above-mentioned technical solution extracts multimodal data such as text, images, and audio from the original data and performs feature fusion. It utilizes convolutional neural network layers to extract local n-gram features, and then uses a Transformer encoder to extract global contextual information. This collaborative approach ensures the comprehensiveness of feature representation and solves the problem of insufficient accuracy in feature representation by traditional single models. Next, efficient and lightweight semantic representation information is generated through distillation optimization. Finally, during decoding, a pointer-generation network mechanism is used, flexibly employing selection (generation) or copying (pointer) methods to effectively solve the out-of-vocabulary (OOV) word problem. While maintaining generalization, it ensures coverage and accuracy, thus achieving efficient, accurate, and robust function word extraction.

[0022] Optionally, step S30 includes: S31, Using multiple convolution kernels of different sizes W (k) For fused feature vectors Convolution processing is performed, expressed as follows: in, Let L represent the matrix dimension of the fused feature vector E, L represent the sequence length, and D represent the feature dimension. W represents the local feature information extracted by a convolutional kernel of size k×D. (k) This represents a convolution kernel of size k×D, b (k) The bias term, E, represents the convolution kernel of size k×D. i:i+k-1 ReLU(.) represents a local k-gram segment of length k starting from position i in the fused feature vector E, and ReLU(.) represents the activation function. S32, For multiple convolution kernels W of different sizes(k) The extracted local feature information is integrated to generate local n-gram feature information F. CNN It can be expressed as the following formula: F CNN =Concat(Pooling(F (3) Pooling(F) (5) ),……) Here, Concat(.) represents concatenation operation and Pooling(.) represents pooling operation; Step S40 includes: S41. The fused feature vector E and the local n-gram feature information F... CNN Feature integration is performed to generate the initial input vector H of the Transformer encoder. (0) ; S42. Calculate the initial input vector H using a multi-head self-attention mechanism. (0) The correlation between any two positions in the data is used to extract global context information; S43. The output vector H after multi-layer Transformer encoding... (Final) R as text semantic representation data Semantic .

[0023] Using the above technical solution, step S30 performs convolution processing using multiple convolution kernels of different sizes and integrates the extracted local feature information to capture local n-gram feature information of different granularities. Step S40 integrates the fused feature vector with the local n-gram feature information and uses a multi-head self-attention mechanism to calculate the correlation between any two positions to extract global context information. This collaborative approach of first extracting local features and then capturing global context provides structured local information for capturing global dependencies, avoids information loss, ensures the comprehensiveness of feature representation, and thus solves the problem of insufficient feature representation accuracy in traditional single models.

[0024] Optionally, step S50 includes: S51. Perform data reasoning based on the text semantic representation data to generate soft target data; S52. Based on the text semantic representation data, perform imitation reasoning to generate student imitation data; S53. Calculate the difference between the student imitation data and the soft target data to determine the distillation loss; S54. Minimize the weighted sum of the distillation loss and the hard target loss through iterative training to generate lightweight semantic representation information.

[0025] Using the above technical solution, step S50 uses a large teacher model to perform data reasoning to generate knowledge-rich soft target data, which then guides a compact and computationally efficient lightweight student model to perform imitation reasoning. By calculating the difference between the soft target data and the student imitation data to determine the distillation loss, and by iteratively training to minimize the weighted sum of the distillation loss and the hard target loss, the "knowledge" of the large teacher model can be efficiently transferred to the lightweight student model. This significantly improves the model's inference speed and deployment efficiency while maintaining high performance, ultimately generating efficient lightweight semantic representation information.

[0026] Optionally, the efficacy word extraction method based on multimodal adaptive learning further includes: S80. Dynamically update the model parameters used in steps S30 to S70 through incremental learning; S90. Generate a visual interface to provide explanatory information to the user, including: S91. When generating or copying efficacy words, receiving contextual attention weight information corresponding to the efficacy words; S92. When the hybrid encoder generates text semantic representation data, receiving self-attention weight information corresponding to the text semantic representation data; S93. Generate an explanatory data package based on the efficacy words, the contextual attention weight information, the text semantic representation data, and the self-attention weight information; S94. Generate a visual interface based on the explanatory data package.

[0027] Using the above technical solution, step S80 enables the model to adapt to dynamic text streams through incremental learning, minimizing the forgetting of old knowledge while learning new knowledge, thus ensuring the model's continuous adaptability to new data. Step S90 captures contextual attention and self-attention weight information during the efficacy word extraction process and generates explanatory data packages and visualization interfaces accordingly. This transforms the complex deep learning reasoning process into a user-understandable visual explanation, intuitively demonstrating the decision-making logic of efficacy word extraction, thereby enhancing the system's traceability and user trust.

[0028] In summary, this application includes at least one of the following beneficial technical effects: 1. The input module receives multimodal data including text, images, and audio, and the preprocessing module performs feature fusion. Utilizing a hybrid encoder module, local n-gram features are first extracted through a convolutional neural network layer, followed by global contextual information extraction through a Transformer encoder. This collaborative mechanism ensures the comprehensiveness of feature representation and overcomes the limitations of traditional single models in feature representation accuracy. Subsequently, the knowledge distillation module optimizes the text semantic representation data, generating efficient and lightweight semantic representation information. Finally, the decoder module integrates a vocabulary and employs a strategy combining generation (matching preset words) and copying (not matching preset words). This essentially utilizes the idea of ​​a pointer-generation network, effectively solving the out-of-vocabulary (OOV) word problem. While maintaining generalization, it ensures coverage and accuracy, thus achieving efficient, accurate, and robust functional word extraction. Comparative experiments verify that the F1 score of the proposed model (90.3%) is significantly higher than that of the baseline model (79.5%) and the contrasting model (86.1%). When handling out-of-vocabulary (OOV) words, the F1 score of our model (78.6%) is approximately 149% higher than the comparison model (31.5%). Furthermore, our model is optimized through a knowledge distillation module, reducing the number of parameters to approximately 42M, a reduction of approximately 61.8%. Simultaneously, the inference latency is reduced from 48ms to 21ms, resulting in an inference speed improvement of approximately 128%.

[0029] 2. Convolutional neural network layers use multiple convolutional kernels of different sizes to convolve the fused feature vector and integrate the extracted local feature information, enabling the capture of local n-gram feature information at different granularities. The Transformer encoder then integrates the fused feature vector with this local n-gram feature information and uses a multi-head self-attention mechanism to calculate the correlation between any two positions, thereby capturing long-range dependencies and global contextual information in the sequence. This collaborative approach of CNN extracting local features and Transformer capturing global dependencies provides structured local information for the global dependency capture unit, avoiding information loss and ensuring the comprehensiveness of feature representation, thus solving the problem of insufficient feature representation accuracy in traditional single models.

[0030] 3. The knowledge distillation module uses a high-performance, large teacher model unit to infer from the textual semantic representation data, generating knowledge-rich soft-objective data. Simultaneously, a compact and computationally efficient lightweight student model unit performs imitation inference on the same textual semantic representation data, generating student imitation data. The distillation learning and optimization unit determines the distillation loss by calculating the difference between the two models and minimizes the weighted sum of the distillation loss and the hard-objective loss. This approach successfully transfers the "knowledge" of the large teacher model to the lightweight student model, significantly improving the model's inference speed and deployment efficiency while maintaining high performance as much as possible, ultimately generating efficient lightweight semantic representation information.

[0031] 4. By adding an incremental learning module, the system can collect new data, user feedback, and error cases encountered in the production environment, and use this data to dynamically update the model parameters of the hybrid encoder, knowledge distillation, and decoder modules through incremental learning. This enables the model to adapt to dynamic text streams, minimizing the forgetting of old knowledge while learning new knowledge, thus ensuring the model's continuous adaptability to new data. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of a module of a multimodal adaptive learning-based efficacy word extraction system according to an embodiment of this application; Figure 2 This is a schematic diagram of the input module in an embodiment of this application; Figure 3 This is a schematic diagram of the preprocessing module in an embodiment of this application; Figure 4 This is a schematic diagram of the hybrid encoder module according to an embodiment of this application; Figure 5 This is a schematic diagram of the knowledge distillation module according to an embodiment of this application; Figure 6 This is a schematic diagram of the decoder module according to an embodiment of this application; Figure 7 This is a schematic diagram of the output module in an embodiment of this application; Figure 8 This is a schematic diagram of the explanatory modules of an embodiment of this application; Figure 9 This is a schematic diagram of the incremental learning module in an embodiment of this application; Figure 10 This is a flowchart illustrating a function word extraction method based on multimodal adaptive learning according to an embodiment of this application.

[0033] Figure labeling: 10, Input module; 11, Data acquisition unit; 12, Data parsing unit; 13, Data caching and forwarding unit; 20, Preprocessing module; 21, Text filtering unit; 22, Multimodal adaptation unit; 23, Feature fusion unit; 30, Hybrid encoder module; 31, Local feature extraction unit; 32, Global dependency capture unit; 40, Knowledge distillation module; 41, Teacher model unit; 42, Student model unit; 43, Distillation learning and optimization unit; 50, Decoder module; 51, Pointer-Generator network unit; 52, Domain Adaptation unit; 53, Decoding output and correction unit; 60, Output module; 61, Result formatting unit; 62, Data distribution and interface unit; 70, Interpretation module; 71, Attention weight capture unit; 72, Visualization generation unit; 80, Incremental learning module; 81, Data and feedback collection unit; 82, Knowledge maintenance and conflict detection unit; 83, Model update unit. Detailed Implementation

[0034] The following combination Figures 1-10 This application will be described in further detail.

[0035] This application discloses a efficacy word extraction system based on multimodal adaptive learning.

[0036] Figure 1 This is a schematic diagram of a module of a efficacy word extraction system based on multimodal adaptive learning according to an embodiment of this application. (Refer to...) Figure 1 The efficacy word extraction system includes an input module 10, a pre-processing module 20, a hybrid encoder module 30, a knowledge distillation module 40, a decoder module 50, an output module 60, an interpretability module 70, and an incremental learning module 80.

[0037] The input module 10 is connected to the preprocessing module 20, which in turn connects to the hybrid encoder module 30, the knowledge distillation module 30, the decoder module 50, the output module 60, and the interpretability module 70. The incremental learning module 80 connects the hybrid encoder module 30, the knowledge distillation module 40, and the decoder module 50, achieving efficient and accurate extraction of functional words and adaptability to dynamic text stream updates. This is because the modules collaborate with each other. The input module 10 provides raw data from different data sources (e.g., document text, product descriptions, social media posts and their accompanying images and audio). The preprocessing module 20 preprocesses the raw data, including data cleaning and feature fusion. The hybrid encoder module 30 performs deep encoding on the preprocessed data to generate a text semantic representation. The knowledge distillation module 40 distills and optimizes the text semantic representation. The decoder module 50 decodes the distilled and optimized semantic representation to extract efficacy words. A Pointer-Generator Network is used to address the out-of-vocabulary (OOV) problem and enhance generalization ability. The output module 60 outputs the efficacy words extracted by the decoder module 50. The interpretability module 70 provides a visual explanation of the efficacy word extraction logic to enhance user trust. The incremental learning module 80 dynamically updates the model to ensure its adaptability to new data.

[0038] Figure 2 This is a schematic diagram of the input module according to an embodiment of this application. (Refer to...) Figure 2 The input module 10 includes a data acquisition unit 11, a data parsing unit 12, and a data caching and forwarding unit 13.

[0039] The data acquisition unit 11 is responsible for establishing connections with external data sources, which can be connected to databases (such as literature databases), API interfaces (such as cloud service data streams or social media interfaces), or local file systems, etc. It can use interface connections to obtain raw data files or data packets of different formats, or it can use different network interface protocols to connect to external data sources.

[0040] The data parsing unit 12 is connected to the data acquisition unit 11 and can identify and separate the received raw data, determine the modality type of the raw data (text, video, and audio, etc.), and perform structured processing on this raw data. Specifically, the data parsing unit 12 first determines the format of the raw data, such as identifying file formats like .txt, .pdf, .jpg, and .mp3. Next, the data parsing unit 12 performs modality separation on the raw data to generate multimodal data, extracting text information from unstructured or semi-structured documents and separating image and audio data. The data parsing unit 12 also extracts metadata corresponding to the multimodal data. This metadata is the foundation for subsequent feature fusion and incremental learning, ensuring that heterogeneous data can be correctly associated and effectively utilized by the model. The metadata includes, for example, association ID, timestamp, source ID, location information, and modality identifier. The association ID is used to link data from different modalities, for example, to ensure that the image file image_001.jpg and the audio file audio_001.mp3 belong to the same product description text. The timestamp records the data collection time, which helps the incremental learning module 80 determine the data's age. The source ID identifies whether the data comes from a literature library, e-commerce platform, or social media, facilitating the training of the domain adaptation unit 52. Location information includes, for example, the location where the image and audio data were taken / recorded, providing additional contextual information in certain application scenarios (e.g., social media analysis). The modality identifier identifies the modality type of the data (e.g., text, image, or audio), enabling the downstream multimodal adaptation unit 22 to call the correct feature extractor.

[0041] The data caching and forwarding unit 13 is connected to the data parsing unit 12, serving as a buffer between the input module 10 and the preprocessing module 20 to ensure the stability and synchronization of the data stream. Specifically, the data caching and forwarding unit 13 can synchronize associated multimodal data (e.g., associated text information and image data) to ensure that associated multimodal data can be transmitted synchronously in correct batches. It can also temporarily cache large multimodal files to prevent downstream module blockage. Finally, the multimodal data parsed by the data parsing unit 12 and its corresponding metadata are forwarded by the data caching and forwarding unit 13 to the preprocessing module for further filtering and feature extraction.

[0042] Figure 3 This is a schematic diagram of the preprocessing module according to an embodiment of this application. (Refer to...) Figure 3The preprocessing module 20 includes a text filtering unit 21, a multimodal adaptation unit 22, and a feature fusion unit 23.

[0043] The text filtering unit 21 is connected to the input module 10 and is used to perform semantic-level filtering and standardization on the text information to generate text feature information, and to generate text feature vectors from the text feature information. Specifically, the text filtering unit 21 performs data cleaning on the text information, removing stop words, expressions, and special symbols. Then, it performs standardization processing on the cleaned text information, including lemmatization, stemming, and case conversion. Furthermore, the text filtering unit 21 can also identify and remove redundant or low-information sentences through semantic similarity calculation based on word embeddings.

[0044] The multimodal adaptation unit 22 is connected to the input module 10 and is used to extract visual feature vectors and auditory feature vectors from image data and audio data in multimodal data. Specifically, the multimodal adaptation unit 22 can extract visual features from the image data using a convolutional neural network (CNN) or other visual models (such as ResNet). After performing Mel-frequency cepstral coefficient (MFCC) processing on the audio data, it uses a convolutional neural network or other sequence models to extract auditory features from the audio data. The visual and auditory feature information is then converted into visual and auditory feature vectors compatible with the text feature vectors, respectively.

[0045] The feature fusion unit 23 is connected to the text filtering unit 21 and the multimodal adaptation unit 22, and is used to perform feature fusion on the text feature vector, the visual feature vector, and the auditory feature vector to generate a fused feature vector, and output the fused feature vector to the hybrid encoder module 30. The feature fusion unit 23 can employ early fusion, late fusion, or intermediate fusion strategies, performing operations such as concatenation or weighted summation on the text feature vector, the visual feature vector, and the auditory feature vector to generate the fused feature vector.

[0046] Figure 4 This is a schematic diagram of the hybrid encoder module according to an embodiment of this application. (Refer to...) Figure 4 The hybrid encoder module 30 includes a Local Feature Extraction Unit 31 and a Global Dependency Capturing Unit 32.

[0047] The local feature extraction unit 31 is connected to the preprocessing module 20 and mainly consists of a convolutional neural network layer (CNN layer). It is responsible for efficiently extracting local n-gram features from the fused feature vector. The CNN layer can use multi-scale convolutional kernels to capture local information of different granularities. It receives the fused feature vector from the preprocessing module 20 and quickly identifies and extracts local feature information of local patterns such as words and phrases through sliding window and convolution operations. Then, these local feature information are transmitted to the global dependency capture unit 32 through pooling or direct feature mapping.

[0048] Specifically, the fused feature vector E input to the local feature extraction unit 31 can be expressed as follows: in, Let E represent the matrix dimension of the fused feature vector, L represent the sequence length, and D represent the feature dimension.

[0049] For the fused feature vector E, the local feature extraction unit 31 uses convolution kernels W of different sizes. (k) The fused feature vector E is convolved, as shown in the following equation: in, W represents the local feature information extracted by a convolutional kernel of size k. (k) This represents a convolution kernel of size k×D, b (k) The bias term, E, represents the convolution kernel of size k×D. i:i+k-1 ReLU(.) represents a local k-gram segment of length k starting from position i in the fused feature vector E, and ReLU(.) represents the activation function.

[0050] Then, the local feature information extracted by convolutional kernels of all sizes is integrated to generate the final local n-gram feature information F. CNN This step can be achieved through feature concatenation, expressed as follows: F CNN =Concat(Pooling(F (3) Pooling(F) (5) ),……) Here, Concat(.) represents concatenation operation and Pooling(.) represents pooling operation.

[0051] The global dependency capture unit 32 is mainly composed of a Transformer encoder, used to capture long-distance dependencies and global context information in the entire input sequence (including fused feature vectors and local feature information). The Transformer encoder includes a multi-head self-attention mechanism (MHSA) and a feedforward network. It receives the fused feature vector from the preprocessing module 20 and the local feature vector from the local feature extraction unit 31. Through the MHSA mechanism, it calculates the correlation between any two positions in the input sequence, thereby understanding the global semantics and contextual relationships. The feature representation enhanced by the global context is used as the final output text semantic representation data.

[0052] Specifically, the global dependency capture unit 32 first processes the fused feature vector E and the local n-gram feature information F. CNN The integration is expressed as follows: H (0) =E+W proj F CNN Among them, H (0) Represents the initial input vector of the Transformer encoder, W proj Represents the projection matrix, used to project F CNNAlign the dimensions of E to the dimensions of E, and then perform element-wise addition.

[0053] Then, the initial input vector H is adjusted using a learnable weight matrix. (0) A linear transformation is performed to generate the attention weight vector, expressed as follows: Q = H (0) W Q K = H (0) W K V = H (0) W V Where Q represents the query vector, K represents the key vector, V represents the value vector, and W represents the value vector. Q Represents the query weight matrix, W K Represents the key weight matrix, W V The Transformer encoder then calculates attention weights to determine the correlation between any two positions in the initial input vector, expressed as follows: Where Attention(Q,K,V) represents the attention weight matrix, d k The dimension of the key vector is represented by , and softmax(.) represents the normalization function. Each element A in the attention weight matrix... ij When processing the i-th token, how much attention should be given to the j-th token?

[0054] Furthermore, the outputs of the H attention heads are concatenated and linearly transformed using a multi-head self-attention mechanism, as shown in the following equation: MHSA(H (l) = Concat(Head1, ...,Head) H W O Among them, H (l) Represents the input features of the l-th layer of the Transformer encoder, Head H W represents the output of the Hth attention head (i.e., the attention weights calculated by the Hth attention head). O This represents the output weight matrix.

[0055] Features processed by the multi-head attention mechanism are then further processed by residual connections, layer normalization, and feedforward networks (FFN) to obtain... H (l+1) =LayerNorm(H(l) +FFN(LayerNorm(H (l) +MHSA(H (l) )))) Among them, H (l+1) This represents the output feature of the (l+1)th layer of the Transformer encoder.

[0056] After multiple layers of Transformer encoding, the final output H is obtained. (Final) This is the final text semantic representation data.

[0057] In summary, the hybrid encoder module 30 first extracts local features through the local feature extraction unit 31, providing structured local information for the global dependency capture unit 32. The global dependency capture unit 32 then captures the global context based on this information. The two work together to avoid information loss and ensure the comprehensiveness of feature representation, thereby solving the problem of insufficient feature representation accuracy in traditional single models.

[0058] Figure 5 This is a schematic diagram of the knowledge distillation module according to an embodiment of this application. (Refer to...) Figure 5 The knowledge distillation module 40 includes a teacher model unit 41, a student model unit 42, and a distillation learning and optimization unit 43.

[0059] The teacher model unit 41 includes a pre-trained, large-parameter, and high-performance large teacher model, which can be a deep and complex neural network model (e.g., BERT-large or other high-precision models). The teacher model unit 41 receives text semantic representation data from the hybrid encoder module 30. During the training phase, it infers from the text semantic representation data and generates soft targets or intermediate layer representations with rich semantic information. These soft targets (typically probability distributions after softmax) contain the teacher model's knowledge of the text semantic representation data and are used to guide the training of the student model. Specifically, the teacher model first infers from the input text semantic representation data and outputs the original logits vector; then, a temperature parameter T is introduced to soften the logits, retaining fine-grained semantic associations by adjusting the smoothness of the probability distribution. Regarding the temperature parameter setting strategy, a dynamic annealing mechanism is adopted: In the initial stage (first 10% of iterations), a higher temperature (T=10) is used to capture the global probability distribution characteristics of the teacher model for complex semantic scenarios, helping the student model quickly learn implicit associations within the domain; in the mid-term stabilization stage (middle 80% of iterations), the temperature is linearly reduced to T=5, gradually focusing on the refined transmission of key knowledge; in the convergence stage (last 10% of iterations), T=5 is fixed to ensure that the output probability distribution of the student model is consistent with that of the teacher model. Through this dynamic adjustment, soft target data can both retain the teacher model's semantic reasoning ability for out-of-vocabulary words and technical terms, and avoid the blurring of decision boundaries caused by excessive softening.

[0060] The student model unit 42 includes a compact and computationally efficient lightweight student model, which can be a simplified version of the teacher model (e.g., DistilBERT or other models that have been pruned and quantized). The student model unit 42 receives text semantic representation data from the hybrid encoder module 30 and attempts to mimic the soft target data generated by the teacher model unit 41 during training, outputting student-mimicked data. The inference speed and deployment efficiency of the student model unit 42 are both higher than those of the teacher model unit 41.

[0061] The distillation learning and optimization unit 43 is connected to the teacher model unit 41 and the student model unit 42, and is responsible for executing the knowledge distillation training process. The distillation learning and optimization unit 43 calculates the difference between the soft target data generated by the teacher model unit 41 and the student imitation data generated by the student model unit 42 to determine the distillation loss calculation. This process can use Kullback-Leibler divergence for calculation. During training, the distillation learning and optimization unit 43 can integrate adversarial training strategies, introducing small adversarial perturbations to enable the student model unit 42 to learn more stable features, thereby enhancing the model's robustness to noise. The training objective is to minimize the weighted sum of the distillation loss and the standard hard target loss. Finally, the distillation-optimized lightweight semantic representation information is output to the decoder module 50.

[0062] Specifically, the total loss L calculated by the distillation learning and optimization unit 43 Total Typically caused by hard target loss L Hard And soft target loss (distillation loss) L Soft The weighted composition is expressed as follows: L Total =α·L Soft +(1-α)L Hard Among them, L Hard It is the standard cross-entropy loss calculated by student model unit 42 on the hard target (i.e., the true label), L Soft The output P of student model unit 42 S The soft target P generated by teacher model unit 41 T The difference between them is usually calculated using the Kullback-Leibler (KL) divergence with a temperature coefficient T to smooth the probability distribution and guide student model unit 42 to learn the "knowledge" of teacher model unit 41, expressed as follows: L Soft =KL(softmax(z) T / T)||softmax(z S / T)) Among them, z S and z T These are the logits outputs of student model unit 42 and teacher model unit 41, respectively, where α is a hyperparameter balancing the two loss terms. By minimizing L... Total The student model unit 42 is able to mimic the behavior of the teacher model unit 41, thereby generating efficient and lightweight semantic representation information.

[0063] In summary, the internal workflow of the knowledge distillation module is a three-way game and optimization process that transfers the "knowledge" of a high-performance large model (teacher model unit 41) to a compact lightweight model (student model unit 42) while maintaining performance as much as possible, ultimately generating an efficient lightweight semantic representation information.

[0064] Figure 6 This is a schematic diagram of the decoder module according to an embodiment of this application. (Refer to...) Figure 6 The decoder module 50 includes a pointer-generator network unit 51, a domain adaptation unit 52, and a decoding output and correction unit 53.

[0065] The pointer-generator network unit 51 is connected to the knowledge distillation module 40 and mainly consists of a decoder based on a sequence-to-sequence model. It uses a gating mechanism or a switch to switch between generator mode and pointer mode.

[0066] Specifically, the pointer-generation network unit 51 receives lightweight semantic representation information from the knowledge distillation module 40 and generates predicted words based on this information. When a predicted word exists in a preset vocabulary, the decoder switches to generation mode and selects the word with the highest matching probability from the vocabulary as the output word based on a traditional softmax layer. This mode is suitable for extracting common and general efficacy terms, such as "whitening," "moisturizing," and "antioxidant." These words appear frequently in the training data, and the model can reliably generate them from the vocabulary.

[0067] When the predicted word is an out-of-vocabulary (OOV) word not present in the vocabulary or a specific proper noun, the decoder switches to pointer mode, pointing the "pointer" to a specific position in the lightweight semantic representation information and directly copying the word (token) at that position into the output text sequence. An attention mechanism can be used to calculate the importance score of each word in the lightweight semantic representation information. In pointer mode, attention weights are converted into copy probabilities, and the decoder selects the input word with the highest attention weight for copying. This mode effectively solves the OOV problem and handles out-of-vocabulary words not present in the training dictionary (e.g., new chemical ingredient names, specific product models, or rare technical terms), which are likely to be the efficacy words to be extracted.

[0068] The core of the pointer-generation network unit 51 in solving the OOV problem lies in its strategy of combining generation and copying. Specifically, at each time step t of decoding, the system calculates a generation probability p. gen ∈[0,1]. The probability p gen It can be calculated by a gating unit, whose input typically includes the current hidden state s of the decoder. t and the context vector c at that time step t It can be expressed as the following formula: p gen =σ(W g ·[s t c t ]+b g ) Where σ is the sigmoid activation function, W g and b g These are learnable parameters.

[0069] The probability P(w) of the final output word w is the "generation" probability P. vocab (w) and the probability of "replication" P copy The weighted sum of (w) is expressed as follows: P(w)=p gen ·P vocab (w)+(1-p gen )·P copy (w) Among them, P vocab (w) is the probability that the decoder generates word $w$ from the preset vocabulary, calculated using a traditional softmax layer. copy (w) is the probability that word w is copied from the input sequence (i.e., lightweight semantic representation information), which is directly derived from the attention weight of the input sequence at time step t. (Right now If a word w is an out-of-vocabulary (OOV) word, its P... vocab (w) will be 0, but as long as it appears in the input sequence, the system still has (1-p) gen )·P copy The probability of (w) is used to copy it, thus effectively solving the OOV problem and ensuring the coverage of efficacy word extraction.

[0070] In summary, the pointer-generation network unit 51 achieves efficient and robust efficacy word extraction by dynamically switching between generation mode and pointer mode. When faced with common words, it selects generation mode to maintain generalization, while when faced with new words or specific phrases, it selects pointer mode to ensure coverage and accuracy.

[0071] The domain adaptation unit 52 is connected to the knowledge distillation module 40 to enhance the system's generalization ability across different domain texts (such as patents, e-commerce, and medical texts), enabling cross-domain transfer. The domain adaptation unit 52 is primarily constructed based on the concept of Domain Adversarial Neural Network (DANN), comprising a domain classifier and a gradient reversal layer. During training, the domain classifier calculates the domain classification loss of the lightweight semantic representation information to identify the domain from which the input features originate. The gradient reversal layer inverts the gradient of the domain classification loss and feeds it back to the knowledge distillation module 40, guiding the weight adjustment of the knowledge distillation module 40. This ensures that the distilled and optimized lightweight semantic representation information is domain-independent. Therefore, when encountering new and unseen domain text in a production environment, the features received by the pointer-generation network unit 51 are domain-neutral, enabling better switching between generation and pointer modes and avoiding a sharp performance drop due to domain differences.

[0072] The decoding output and correction unit 53 is connected to the pointer-generation network unit 51 and is used to post-process and correct the decoded output sequence output by the pointer-generation network unit 51, cleaning up any duplicate words, incomplete phrases, or punctuation marks that may exist in the decoded output sequence, and generating a list of efficacy words. Finally, the decoding output and correction unit 53 forwards the list of efficacy words to the output module 60 and forwards the attention weights from the decoding process to the interpretability module 70.

[0073] Figure 7 This is a schematic diagram of the output module according to an embodiment of this application. (Refer to...) Figure 7 The output module 60 includes a result formatting unit 61 and a data distribution and interface unit 62.

[0074] The result formatting unit 61 is connected to the decoder module 50 and is used to receive the list of efficacy terms generated by the decoder module 50 and convert it into a standard, unified output format, such as JSON, XML or a simple list / table. It also associates the efficacy terms with relevant metadata to ensure that the output efficacy terms are consistent with the efficacy dictionary of the enterprise or industry standards.

[0075] The data distribution and interface unit 62 is responsible for transmitting the formatted final result to the target recipient. According to the recipient's needs, it distributes the data through API, message queue or file system, etc. It also manages the interaction with the cloud deployment interface, supports pushing the result to the distributed storage or real-time analysis platform in real time, and forwards the visualization information containing the extraction logic to the interpretability module 70.

[0076] Figure 8 This is a schematic diagram illustrating the explanatory modules of an embodiment of this application. (Refer to...) Figure 8 The explanatory module 70 is used to transform the complex deep learning reasoning process into a user-understandable visual explanation, including an attention weight capturing unit 71 and a visualization generation unit 72.

[0077] The attention weight capture unit 71 is connected to the decoder module 50 and the hybrid encoder module 30, and is used to capture and store in real time the attention weights calculated by the model during the efficacy word extraction process from the key layers of the decoder module 50 and the hybrid encoder module 30. Specifically, when the decoder module 50 generates (or copies) each efficacy word, the attention weight capture unit 71 receives the corresponding context attention weights, which indicate which words in the input text contribute the most to the current output. The attention weight capture unit 71 also collects self-attention weights from the Transformer encoder of the hybrid encoder module 30 to demonstrate how the model understands the internal relationships within the text. Then, the attention weight capture unit 71 associates the captured context attention weights and self-attention weights with the original text semantic representation data and the extracted efficacy word list, respectively, and stores them in a structured manner as an Interpretability Data Package.

[0078] The visualization generation unit 72 is connected to the attention weight capture unit 71 and is used to generate an intuitive and easy-to-understand visualization interface using structured interpretive data packages. The visualization generation unit 72 can render and generate an attention heatmap based on the interpretive data packages, where the captured attention weight information is mapped onto the original text. For example, words with darker colors (or higher highlight levels) indicate that the model gives them more attention when extracting efficacy words, thus intuitively demonstrating the contribution of the extraction decision. For complex extraction processes, the visualization generation unit 72 can also generate a simplified decision path diagram, illustrating whether the model selected a generation mode or a pointer mode, further enhancing traceability.

[0079] Figure 9 This is a schematic diagram of the incremental learning module according to an embodiment of this application. (Refer to...) Figure 9 The incremental learning module 80 includes a data and feedback collection unit 81, a knowledge maintenance and conflict detection unit 82, and a model updating unit 83.

[0080] The data and feedback collection unit 81 is responsible for collecting new data encountered by the system in the production environment, user-corrected labeled data, and error cases or low-confidence results where the model performs poorly in actual inference. It continuously monitors new data streams from the input module 10 and real-time feedback from the output module 60. It can also receive high-quality data corrected and relabeled by human annotators or domain experts through the Manual Verification Interface. After preliminary filtering and deduplication, the collected data is transported to the knowledge maintenance and conflict detection unit 82.

[0081] The knowledge maintenance and conflict detection unit 82 is connected to the data and feedback collection unit 81 and is used to manage the model's knowledge base and assess the potential impact or conflict of new data on existing knowledge. The knowledge maintenance and conflict detection unit 82 stores a portion of representative historical data samples. When training with new data, it mixes the new data with historical data and uses regularization or gradient analysis techniques to evaluate whether the training gradient of the new data conflicts with the key gradient directions of the historical data. Different training weights are assigned based on the data's age, quality, and the degree of impact on existing knowledge, thereby generating an incremental training dataset.

[0082] The model update unit 83 is connected to the knowledge maintenance and conflict detection unit 82 and is used to receive the incremental training dataset, perform actual incremental training and parameter updates of the model, and minimize the learning cost of new knowledge and the forgetting of old knowledge. Specifically, the model update unit 83 uses the incremental training dataset to perform mini-batch iterative training on the system's main model (including the hybrid encoder module 30, knowledge distillation module 40, and decoder module 50), and typically only fine-tunes a portion of the model's parameter layers or updates all parameters with a small learning rate to limit the magnitude of change. The performance of the updated model is evaluated on a small, retained validation set. If the performance meets the requirements, the new model weights are deployed back to the main system, completing the dynamic update of the model.

[0083] To verify the technical effectiveness of the embodiments of this application, this application provides a set of comparative experiments based on a public dataset (e.g., a multimodal product review dataset from an e-commerce platform). The baseline model adopts a scheme similar to that in invention patent application publication number CN120821890A, using multimodal fusion features, but only extracting local features through CNN before decoding. The comparative model simulates the scheme in invention patent application publication number CN118797038A, using a hybrid encoder (CNN+Transformer), but excluding the knowledge distillation module and pointer-generation network decoder; the decoder uses standard Softmax. The model in this application adopts the complete technical solution of this application, including a hybrid encoder, a knowledge distillation module, and a pointer-generation decoder. Experimental metrics include efficacy word extraction accuracy and model efficiency. Efficacy word extraction accuracy is evaluated using the F1 score, and the F1 score for out-of-vocabulary (OOV) words is calculated separately. Model efficiency is compared by the number of model parameters and the average inference latency per sample. The experimental results are shown in Tables 1 and 2.

[0084] Table 1: Comparison of the accuracy of efficacy keyword extraction Model F1 score (Overall) F1 score (OOV) Baseline model 79.5％ 18.2％ Contrast Model 86.1％ 31.5％ This application model 90.3％ 78.6％ Table 2: Model Efficiency Comparison Model Number of parameters Inference latency (ms / sample) Contrast Model 110M 48ms This application model 42 M 21ms As shown in Table 1, the F1 score of our model (90.3%) is significantly higher than that of the baseline model (79.5%) and the comparative model (86.1%). This is attributed to the hybrid encoder's collaborative capture of local n-gram features and global contextual information. More importantly, when dealing with out-of-vocabulary (OOV) words, our model's F1 score (78.6%) is approximately 149% higher than that of the comparative model (31.5%). This fully demonstrates the significant advantage of the pointer-generative network decoder in solving the OOV problem, ensuring the coverage and accuracy of effective word extraction. As shown in Table 2, the comparative model has a large number of parameters (approximately 110M) due to the use of the standard Transformer. Our model, however, is optimized through a knowledge distillation module, reducing the number of parameters to approximately 42M, a reduction of approximately 61.8%. Simultaneously, the inference latency is reduced from 48ms to 21ms, representing an improvement in inference speed of approximately 128%. This demonstrates that this application has successfully generated efficient and lightweight semantic representation information without sacrificing (or even improving) accuracy, solving the problem of large and difficult-to-deploy related technical models.

[0085] The implementation principle of a multimodal adaptive learning-based efficacy word extraction system according to an embodiment of this application is as follows: Input module 10 receives multimodal data such as text, images, and audio, and preprocessing module 20 performs feature fusion. Using hybrid encoder module 30, local n-gram features are first extracted through convolutional neural network layers, and then global contextual information is extracted through Transformer encoder. This collaborative mechanism ensures the comprehensiveness of feature representation and overcomes the limitations of traditional single models in feature representation accuracy. Subsequently, knowledge distillation module 40 optimizes the text semantic representation data to generate efficient and lightweight semantic representation information. Finally, decoder module 50 integrates a vocabulary and adopts a strategy combining generation (matching preset words) and copying (not matching preset words). This essentially utilizes the idea of ​​pointer-generation networks, effectively solving the out-of-vocabulary (OOV) problem, ensuring coverage and accuracy while maintaining generalization, thereby achieving efficient, accurate, and robust efficacy word extraction.

[0086] This application also provides a method for extracting efficacy words based on multimodal adaptive learning, which is mainly implemented by the efficacy word extraction system based on multimodal adaptive learning in the above embodiments.

[0087] Figure 10 This is a flowchart illustrating a function word extraction method based on multimodal adaptive learning according to an embodiment of this application. (Refer to...) Figure 10 The method includes the following steps: S10. Extract multimodal data from the original data, wherein the multimodal data includes text information, image data and audio data.

[0088] S20. Perform feature fusion on the multimodal data to generate a fused feature vector.

[0089] S30. Extract local n-gram feature information from the fused feature vector through a convolutional neural network layer.

[0090] S40. Extract global context information from the fused feature vector and the local n-gram feature information using a Transformer encoder to generate text semantic representation data.

[0091] S50. Distillation optimization is performed on the text semantic representation data to generate lightweight semantic representation information.

[0092] S60. Generate a prediction word based on the lightweight semantic representation information.

[0093] S70. Match the predicted word with preset words in the vocabulary. If a preset word that matches the predicted word exists, select the predicted word as the efficacy word. If no preset word that matches the predicted word exists, copy the efficacy word from the lightweight semantic representation information.

[0094] S80. Dynamically update the model parameters used in steps S30 to S70 through incremental learning.

[0095] S90. Generate a visual interface to provide explanatory information to users.

[0096] The steps of this method are explained in detail below.

[0097] In step S10, multimodal data is extracted from the raw data. The multimodal data includes text information, image data, and audio data. This step is mainly implemented by the input module 10.

[0098] In step S20, feature fusion is performed on the multimodal data to generate a fused feature vector. This step is mainly implemented by the preprocessing module 20.

[0099] In step S30, local n-gram feature information is extracted from the fused feature vector using a convolutional neural network layer. In step S40, global context information is extracted from the fused feature vector and the local n-gram feature information using a Transformer encoder to generate text semantic representation data. Steps S30 and S40 are primarily implemented by the hybrid encoder module 30.

[0100] Specifically, step S30 includes: S31, Using multiple convolution kernels of different sizes W (k) For fused feature vectors Convolution processing is performed, expressed as follows: in, Let L represent the matrix dimension of the fused feature vector E, L represent the sequence length, and D represent the feature dimension. W represents the local feature information extracted by a convolutional kernel of size k×D. (k) This represents a convolution kernel of size k×D, b (k) The bias term, E, represents the convolution kernel of size k×D. i:i+k-1 ReLU(.) represents a local k-gram segment of length k starting from position i in the fused feature vector E, and ReLU(.) represents the activation function.

[0101] S32, For multiple convolution kernels W of different sizes (k)The extracted local feature information is integrated to generate local n-gram feature information F. CNN It can be expressed as the following formula: F CNN =Concat(Pooling(F (3) Pooling(F) (5) ),……) Here, Concat(.) represents concatenation operation and Pooling(.) represents pooling operation.

[0102] Step S40 includes: S41. The fused feature vector E and the local n-gram feature information F... CNN Feature integration is performed to generate the initial input vector H of the Transformer encoder. (0) .

[0103] S42. Calculate the initial input vector H using a multi-head self-attention mechanism. (0) The correlation between any two positions in the data is used to extract global context information.

[0104] S43. The output vector H after multi-layer Transformer encoding... (Final) R as text semantic representation data Semantic .

[0105] In step S50, the text semantic representation data is distilled and optimized to generate lightweight semantic representation information. This step is mainly implemented by the knowledge distillation module 40.

[0106] Specifically, step S50 includes: S51. Perform data reasoning based on the text semantic representation data to generate soft target data.

[0107] S52. Based on the text semantic representation data, perform imitation reasoning to generate student imitation data.

[0108] S53. Calculate the difference between the student imitation data and the soft target data to determine the distillation loss.

[0109] S54. Minimize the weighted sum of the distillation loss and the hard target loss through iterative training to generate lightweight semantic representation information.

[0110] In step S60, the generation probability and copying probability are calculated based on the lightweight semantic representation information. In step S70, the generation mode and copying mode are dynamically switched based on the generation probability and the copying probability to generate efficacy words; wherein, if the generation mode is selected, the efficacy words are selected from preset words in the vocabulary; if the copying mode is selected, the efficacy words are copied from the lightweight semantic representation information through a pointer mechanism. Steps S60 and S70 are mainly implemented by the decoder module 50.

[0111] In step S80, the model parameters used in steps S30 to S70 are dynamically updated through incremental learning. This step is mainly implemented by the incremental learning module 80 in conjunction with the input module 10, the output module 60, the hybrid encoder module 30, the knowledge distillation module 40, and the decoder module 50.

[0112] In step S90, a visual interface is generated to provide explanatory information to the user. This step is mainly implemented by the explanatory module 70.

[0113] Specifically, step S90 includes: S91. When generating or copying a function word, receive contextual attention weight information corresponding to the function word.

[0114] S92, The hybrid encoder generates text semantic representation data and receives self-attention weight information corresponding to the text semantic representation data.

[0115] S93. Generate an explanatory data package based on the efficacy words, the contextual attention weight information, the text semantic representation data, and the self-attention weight information.

[0116] S94. Generate a visual interface based on the explanatory data package.

[0117] The implementation principle of the efficacy word extraction method based on multimodal adaptive learning in this application is as follows: Multimodal data such as text, images, and audio are extracted from the original data and feature fusion is performed. Local n-gram features are extracted using convolutional neural network layers, and global contextual information is extracted using a Transformer encoder. This collaborative approach ensures the comprehensiveness of feature representation and solves the problem of insufficient accuracy in feature representation by traditional single models. Next, efficient and lightweight semantic representation information is generated through distillation optimization. Finally, during decoding, a pointer-generation network mechanism is used, flexibly employing selection (generation) or copying (pointer) methods to effectively solve the out-of-vocabulary (OOV) word problem. While maintaining generalization, coverage and accuracy are ensured, thus achieving efficient, accurate, and robust efficacy word extraction.

[0118] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A multi-modal adaptive learning based efficacy word extraction system, characterized in that, The method comprises the following steps: An input module is configured to receive raw data and extract multi-modal data from the raw data, wherein the multi-modal data comprises text information, image data and audio data; A preprocessing module is connected to the input module and is configured to perform feature fusion on the multi-modal data to generate a fusion feature vector; A hybrid encoder module is connected to the preprocessing module, wherein the hybrid encoder module comprises a convolutional neural network layer and a Transformer encoder, the convolutional neural network layer is configured to extract local n-gram feature information from the fusion feature vector, and the Transformer encoder is configured to extract global context information from the fusion feature vector and the local n-gram feature information to generate text semantic representation data; A knowledge distillation module is connected to the hybrid encoder module and is configured to perform distillation optimization on the text semantic representation data to generate lightweight semantic representation information; A decoder module is connected to the knowledge distillation module, wherein the decoder module is integrated with a vocabulary bank; the decoder module is configured to calculate a generation probability and a copy probability based on the lightweight semantic representation information, and dynamically switch between a generation mode and a copy mode based on the generation probability and the copy probability to generate an efficacy word; If the generation mode is selected, the efficacy word is selected from preset words in the vocabulary bank; if the copy mode is selected, the efficacy word is copied from the lightweight semantic representation information through a pointer mechanism. 2.The efficacy term extraction system based on multi-modal adaptive learning according to claim 1, wherein, The preprocessing module is configured to perform feature fusion on the multi-modal data to generate a fusion feature vector; wherein the preprocessing module is configured to perform semantic-level filtering and standardization processing on the text information to generate text feature information; convert the text feature information into a text feature vector; extract visual feature information from the image data; convert the visual feature information into a visual feature vector; perform mel-frequency cepstrum coefficient processing on the audio data and extract auditory feature information; convert the auditory feature information into an auditory feature vector; and perform feature fusion on the text feature vector, the visual feature vector and the auditory feature vector to generate the fusion feature vector. 3.The efficacy term extraction system based on multi-modal adaptive learning according to claim 2, wherein, The convolutional neural network layer is configured to extract local n-gram feature information from the fusion feature vector; wherein the convolutional neural network layer is configured as follows: using multiple different sized convolution kernels W (k) to the fused feature vector convolution processing, denoted by the following equation: wherein, denotes the matrix dimension of the fusion feature vector E, L denotes the sequence length, D denotes the feature dimension, denotes the local feature information extracted by the convolution kernel with size k x D, W (k) denotes the convolution kernel with size k x D, b (k) denotes the bias term of the convolution kernel with size k x D, E i:i+k-1 denotes the local k-gram segment with length k in the fusion feature vector E starting from position i, ReLU(.) denotes the activation function; a plurality of different size convolution kernels W (k) The extracted local feature information is integrated to generate local n-gram feature information F CNN is expressed as follows: F CNN = Concat(Pooling(F (3) ), Pooling(F (5) ),...) wherein Concat(.) represents a concatenation operation and Pooling(.) represents a pooling operation; The Transformer encoder is used to extract global context information from the fusion feature vector and the local n-gram feature information to generate text semantic representation data; wherein the Transformer encoder is configured to: perform feature integration on the fusion feature vector E and the local n-gram feature information F CNN to generate an initial input vector H of the Transformer encoder (0) ; calculate the relevance of any two positions in the initial input vector H through a multi-head self-attention mechanism, thereby extracting global context information; and output a vector H after multi-layer Transformer encoding as text semantic representation data R (0) (Final) Semantic .​​ 4.The multi-modal self-adaptive learning based efficacy term extraction system of claim 1, wherein, The knowledge distillation module comprises a teacher model unit, a student model unit and a distillation learning and optimization unit; the teacher model unit is configured to perform data reasoning based on the text semantic representation data to generate soft target data; the student model unit is configured to perform imitative reasoning based on the text semantic representation data to generate student imitative data; and the distillation learning and optimization unit is configured to calculate the difference between the student imitative data and the soft target data to determine a distillation loss, and minimize the weighted sum of the distillation loss and a hard target loss through iterative training to generate the lightweight semantic representation information. 5.The multi-modal self-adaptive learning based efficacy term extraction system according to claim 1, wherein, The method further comprises the following steps: An incremental learning module connected with the hybrid encoder module, the knowledge distillation module and the decoder module, configured to dynamically update model parameters of the hybrid encoder module, the knowledge distillation module and the decoder module through incremental learning.

6. The multi-modal self-adaptive learning based efficacy term extraction system according to any one of claims 1-5, characterized in that, Further comprising: An explanation module connected with the decoder and the hybrid encoder, configured to generate a visual interface to provide explanatory information to a user; wherein the explanation module is configured to: receive context attention weight information corresponding to the efficacy word when the decoder generates or copies the efficacy word; receive self-attention weight information corresponding to the text semantic representation data when the hybrid encoder generates the text semantic representation data; generate an explanatory data package according to the efficacy word, the context attention weight information, the text semantic representation data and the self-attention weight information; and generate a visual interface according to the explanatory data package.

7. A multi-modal self-adaptive learning based efficacy word extraction method, characterized in that, Comprising the following steps: S10, extracting multi-modal data from original data, the multi-modal data comprising text information, image data and audio data; S20, performing feature fusion on the multi-modal data to generate a fusion feature vector; S30, extracting local n-gram feature information from the fusion feature vector through a convolutional neural network layer; S40, extracting global context information from the fusion feature vector and the local n-gram feature information through a Transformer encoder to generate text semantic representation data; S50, performing distillation optimization on the text semantic representation data to generate lightweight semantic representation information; S60, calculating generation probability and copying probability according to the lightweight semantic representation information; S70, dynamically switching generation mode and copying mode based on the generation probability and the copying probability to generate an efficacy word; wherein if the generation mode is selected, the efficacy word is selected from a preset word in a vocabulary; if the copying mode is selected, the efficacy word is copied from the lightweight semantic representation information through a pointer mechanism. 8.The multi-modal self-adaptive learning based efficacy term extraction method according to claim 7, characterized in that, Step S30 comprises: S31, using a plurality of different size convolution kernels W (k) to the fused feature vector convolution processing is performed, expressed as follows: wherein, denotes the matrix dimension of the fusion feature vector E, L denotes the sequence length, D denotes the feature dimension, denotes the local feature information extracted by the convolution kernel with size k x D, W (k) denotes the convolution kernel with size k x D, b (k) denotes the bias term of the convolution kernel with size k x D, E i:i+k-1 denotes the local k-gram segment with length k in the fusion feature vector E starting from position i, ReLU(.) denotes the activation function; S32, a plurality of different size convolution kernels W (k) The extracted local feature information is integrated to generate local n-gram feature information F CNN is expressed as follows: F CNN = Concat(Pooling(F (3) ), Pooling(F (5) ),...) wherein Concat(.) represents a concatenation operation and Pooling(.) represents a pooling operation; Step S40 comprises: S41. The fused feature vector E and the local n-gram feature information F... CNN Feature integration is performed to generate the initial input vector H of the Transformer encoder. (0) ; S42, calculate the initial input vector H through a multi-head self-attention mechanism (0) correlation of any two positions, thereby extracting global context information; S43, the output vector H after multi-layer Transformer encoding (Final) as the text semantic representation data R Semantic . 9.The multi-modal self-adaptive learning based efficacy term extraction method according to claim 7, characterized in that, Step S50 comprises: S51, performing data reasoning based on the text semantic representation data to generate soft target data; S52, performing imitation reasoning based on the text semantic representation data to generate student imitation data; S53, calculating the difference between the student imitation data and the soft target data to determine a distillation loss; S54, minimizing the weighted sum of the distillation loss and a hard target loss through iterative training to generate lightweight semantic representation information.

10. The multi-modal self-adaptive learning based efficacy term extraction method according to any one of claims 7-9, characterized in that, Further comprising: S80, dynamically updating model parameters used in steps S30 to S70 through incremental learning; S90, generating a visual interface to provide explanatory information to a user, comprising: S91, receiving context attention weight information corresponding to the efficacy word when the efficacy word is generated or copied; S92, the mixed encoder generates text semantic representation data, receives self-attention weight information corresponding to the text semantic representation data; S93, generates an explanatory data package according to the efficacy word, the context attention weight information, the text semantic representation data and the self-attention weight information; S94, generates a visual interface according to the explanatory data package.

Images