Incremental learning-based self-evolution method and system for multi-mode general-purpose model

Abstract

The embodiment of the invention provides a multi-modal universal model self-evolution method and system based on incremental learning, and the method comprises the steps: receiving a current target task and multi-modal data corresponding to the current target task, and determining a corresponding trained target universal model; extracting data features of the multi-modal data to obtain multi-modal data features, and processing the multi-modal data features and the multi-modal data through a target special model to obtain a target result; historical multi-modal data features corresponding to the historical multi-modal data are obtained, and general-purpose model parameters of the trained target general-purpose model are updated; and monitoring the performance index of the trained target general-purpose model according to a preset monitoring period, and if the performance index corresponding to the current preset monitoring period does not meet the preset performance index, adjusting the trained target general-purpose model according to a preset adjustment mode. According to the method, the defect of insufficient collaboration between general knowledge and special knowledge is effectively overcome, and the collaboration effect and the updating effect of updating according to new knowledge are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a method and system for self-evolution of a multimodal communication model based on incremental learning. Background Art

[0002] Currently, the processing of multimodal data mainly relies on static model architecture. When receiving new tasks, the static model needs to be retrained using the full amount of data, resulting in large consumption of computing resources and the possibility of catastrophic forgetting in the static model.

[0003] In the existing technology, in order to solve the problems of high computing resource consumption and possible catastrophic forgetting, optimization can be performed by training a hybrid model with a fixed architecture. However, the hybrid model with a fixed architecture cannot dynamically adjust the collaborative relationship between the general model and the special model according to the target task, and does not achieve autonomous collaborative optimization of the model structure and parameters. It is difficult to take into account both general and professional requirements. In addition, the cost of retraining the model is high, and the retrained model is prone to forgetting old knowledge. Summary of the Invention

[0004] In response to the problems in the existing technology, the present application provides a self-evolution method and system for a multimodal general-specific model based on incremental learning, which can effectively solve the shortcomings of traditional technology in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improve the synergy effect of the multimodal general-specific model and the update effect based on new knowledge.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a method for self-evolution of a multimodal communication model based on incremental learning, comprising: Receive a current target task and multimodal data corresponding to the current target task, and determine, based on the target task, a trained target-specific model corresponding to the target task to be executed, wherein the target-specific model includes a target-general model and a target-specific model; The data features of the multimodal data are extracted through the target general model to obtain the multimodal data features, and the multimodal data features and the multimodal data are processed through the target specific model to obtain the target result corresponding to the target task; Obtain historical multimodal data features corresponding to the historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data, and the target result; Monitor the performance indicators of the trained target communication model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target communication model according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0006] Furthermore, after receiving the current target task and the multimodal data corresponding to the current target task, the method further includes: Perform data cleaning on the multimodal data corresponding to the current target task to obtain cleaned multimodal data; The cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data.

[0007] Furthermore, the method further includes: processing the multimodal data features and the multimodal data through a target-specific model to obtain a specific feature; Determine the general correlation index between the multimodal data features and the current target task, and the special correlation index between the special features and the current target task based on the fusion mechanism; and determine the general weight corresponding to the multimodal data features and the special weight corresponding to the special features based on the general correlation index and the special correlation index; Based on general weights and special weights, multimodal data features and special features are fused to obtain the target results corresponding to the target task.

[0008] Furthermore, the method further includes: receiving feedback of a current target result for the current multimodal data, and correcting the target result based on the feedback to obtain a corrected target result; Determine the loss value between the corrected target result and the target result through a preset loss function, determine the gradient value of the current target general-purpose model based on the loss value, and determine the update direction of the general-purpose model parameters of the target general-purpose model; Receives a preset learning rate and updates the general model parameters based on the optimization algorithm, gradient value, historical multimodal data features, and the preset learning rate.

[0009] Furthermore, the method further includes: extracting historical multimodal data features from a preset memory library, using the historical multimodal data features as regularization terms of a preset loss function, and updating the loss value and the gradient value; Receive a preset learning rate, update the general model parameters based on the optimization algorithm, the updated gradient value and the preset learning rate, and update the historical multimodal data features in the preset memory library based on the current multimodal data features.

[0010] Furthermore, after monitoring the performance indicators of the trained target communication model according to a preset monitoring period, the method further includes: When the performance indicators meet the preset performance indicators, monitor the changing trend of the performance indicators within multiple consecutive preset monitoring cycles; If the performance indicator meets the preset performance indicator within the preset monitoring period of the consecutive preset number of periods, the self-evolution process of the target communication model is stopped, and the communication model parameters of the current target communication model are saved; If the performance indicator does not meet the preset performance indicator within the preset monitoring period of the consecutive preset number of periods, the trained target communication model is adjusted according to the preset adjustment method.

[0011] In a second aspect, the present application provides a multimodal communication model self-evolution system based on incremental learning, comprising: A first processing module is configured to receive a current target task and multimodal data corresponding to the current target task, and determine, based on the target task, a trained target-specific model corresponding to the target task, wherein the target-specific model includes a target-general model and a target-specific model; The second processing module is used to extract data features of the multimodal data through the target general model to obtain multimodal data features, and process the multimodal data features and the multimodal data through the target specific model to obtain a target result corresponding to the target task; An update module is used to obtain historical multimodal data features corresponding to historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, current multimodal data and target results; The adjustment module is used to monitor the performance indicators of the trained target communication model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, the trained target communication model is adjusted according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0012] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for self-evolution of a multimodal communication model based on incremental learning are implemented.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for self-evolution of a multimodal communication model based on incremental learning.

[0014] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal communication model self-evolution method based on incremental learning.

[0015] It can be seen from the above technical solution that the present application provides a self-evolution method and system for a multimodal communication model based on incremental learning, which innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target communication model corresponding to the target task according to the target task, extracts the data features of the multimodal data through the target general model in the target communication model, obtains the multimodal data features, generates the target result through the target-specific model in the target communication model combining the multimodal data features and the multimodal data, and updates the communication model of the target communication model according to the historical multimodal data features, the current multimodal data and the target result. Parameters, monitor the performance indicators of the trained target general-purpose model according to the preset monitoring cycle. If the performance indicators corresponding to the current preset monitoring cycle do not meet the preset performance indicators, adjust the trained target general-purpose model according to the preset adjustment method. It can dynamically adjust the structure and parameters of the target general-purpose model through incremental learning and collaborative updating of the target general-purpose model, reducing the need for full retraining, while improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-purpose model's ability to extract features from multimodal data and adapt to target tasks, and improving the accuracy and robustness of the target results. This method effectively solves the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improves the synergistic effect of multimodal general-purpose models and the update effect based on new knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 Schematic diagram of a process of self-evolution of a multimodal communication model based on incremental learning in an embodiment of the present application; Figure 2 This is a structural diagram of a multimodal communication model self-evolution system based on incremental learning in an embodiment of the present application; Figure 3 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0018] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0021] With the development of artificial intelligence (AI), the demand for processing multimodal data (such as text, image, and audio data) is growing. Traditional multimodal models often require retraining to process new multimodal data and target tasks. This is not only time-consuming and labor-intensive, but also prone to forgetting old knowledge. Incremental learning allows models to continuously absorb new data and knowledge based on existing knowledge, achieving gradual updates and optimizations. However, the application of incremental learning in multimodal models is still imperfect. There is a lack of a system and method that can effectively coordinate and self-evolve between general and specialized knowledge to adapt to the complex and ever-changing needs of multimodal data processing.

[0022] In order to effectively solve the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improve the synergy effect of multimodal general-specific models and the update effect of updating based on new knowledge, this application provides an embodiment of a multimodal general-specific model self-evolution method based on incremental learning, see Figure 1 The multimodal communication model self-evolution method based on incremental learning specifically includes the following contents: Step S101: receiving a current target task and multimodal data corresponding to the current target task, and determining a trained target communication model corresponding to the target task to be executed based on the target task.

[0023] Among them, the target general model includes the target general model and the target specific model.

[0024] Optionally, this embodiment receives a current target task and multimodal data for the target task, where the multimodal data includes at least two of image data, text data, and audio data. The target task can be analyzed to determine the task type and task characteristics of the target task, thereby determining a trained target-specific model corresponding to the target task, where the target-specific model includes a target-specific model and a target-general model.

[0025] Furthermore, the target task can be encoded by a task encoder to generate a task feature vector, so that a task-specific model for executing the target task can be determined by the task feature vector.

[0026] This embodiment realizes determining a target-specific model capable of executing the target task through the target task, thereby being able to accurately process the multimodal data corresponding to the target task.

[0027] Step S102: extracting data features of the multimodal data through the target general model to obtain multimodal data features, processing the multimodal data features and the multimodal data through the target specific model to obtain the target result corresponding to the target task.

[0028] Optionally, this embodiment extracts data features of multimodal data through a target general model to obtain multimodal data features, wherein the multimodal data features include image features corresponding to image data, text features corresponding to text data, and audio features corresponding to audio data.

[0029] Different feature extraction methods can be used according to the different data types of the multimodal data. For example, image features of image data can be extracted through a convolutional neural network, where the convolution layer and pooling layer of the convolutional neural network can learn features such as edges, textures, and shapes in the image data to generate image features; Extract text data using a recurrent neural network or transformer model to obtain text features. Recurrent neural networks can capture sequence information in text data and obtain text features based on this sequence information. Transformer models can process text in parallel using a self-attention mechanism to generate semantic feature vectors (i.e., text features). The audio data is extracted through an audio feature extraction algorithm to obtain audio features.

[0030] In addition, multimodal data features and multimodal data are processed through target-specific models, so that the multimodal data can be analyzed in a refined manner in combination with the professional domain knowledge and rules corresponding to the target task to obtain the target results corresponding to the target task.

[0031] This embodiment achieves the depth of semantic understanding and domain adaptability of multimodal data, improves the collaborative efficiency of multimodal data features obtained through task-general models and target results obtained through task-specific models, and improves the accuracy and robustness of target result generation.

[0032] Step S103: Acquire historical multimodal data features corresponding to the historical multimodal data, and update the general model parameters of the trained target general model based on the historical multimodal data features, the current multimodal data, and the target result.

[0033] Optionally, this embodiment obtains historical multimodal data features corresponding to the historical multimodal data, wherein the historical multimodal data features are obtained by applying the target communication model to the historical multimodal data and can be stored in a preset memory library.

[0034] The general model parameters of the trained target general model are updated according to the historical multimodal data features, the current multimodal data and the target results, wherein a joint loss function can be determined in combination with the target results, wherein the joint loss function includes a fitting term for the current multimodal data features and a retention constraint term for the historical multimodal data features, thereby optimizing the general model parameters.

[0035] This embodiment achieves the ability to retain historical knowledge during the incremental learning process, prevents catastrophic forgetting, improves the model's adaptability to dynamic multimodal environments, and enhances the accuracy of general-purpose model parameter updates, enabling the target general-purpose model to continuously meet the performance requirements of the target results.

[0036] Step S104: monitoring the performance index of the trained target communication model according to a preset monitoring period. If the performance index corresponding to the current preset monitoring period does not meet the preset performance index, adjusting the trained target communication model according to a preset adjustment method.

[0037] The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0038] Optionally, this embodiment performs performance evaluation on the trained target communication model according to a preset monitoring period (e.g., weekly or monthly), where the performance indicators include accuracy, recall rate, F1 value, inference time, etc. If the performance indicators in the current monitoring period do not reach the preset performance indicators, the adjustment mechanism will be automatically triggered.

[0039] In addition, when it is monitored that the performance indicator corresponding to the current preset monitoring period does not reach the preset performance indicator threshold, the trained target communication model is adjusted according to the preset adjustment method, wherein the preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0040] Among them, adjusting the model structure involves modifying the target general module and / or target professional model of the trained target general-purpose model, such as by increasing or decreasing the number of network layers and optimizing the design of the fusion mechanism to adapt to the new data distribution; adjusting the parameter update frequency includes but is not limited to changing the learning rate of the optimization algorithm, adjusting the gradient calculation batch size or introducing a knowledge retention mechanism to control the parameter update step size to ensure the balance of old knowledge retention in the process of absorbing new knowledge. The adjustment process is dynamically executed based on the evaluation results until the performance indicators meet the preset requirements or the evolution stop conditions are reached.

[0041] Furthermore, performance metrics can be calculated by comparing the performance of the target general-purpose model on the validation dataset.

[0042] This embodiment realizes the adaptability and robustness of the target general-purpose model in incremental learning, improves the performance and efficiency of multimodal data processing tasks, and improves the self-evolution stability and generalization capability.

[0043] This embodiment enables the target-generalized model to share knowledge between the task-general model and the task-specific model, improves generalization capabilities, continuously optimizes itself based on new multimodal data, adapts to environmental changes, and enables the target-generalized model to operate stably in the long term through performance monitoring and dynamic adjustment.

[0044] In some embodiments, after receiving the current target task and the multimodal data corresponding to the current target task, the method further includes: Perform data cleaning on the multimodal data corresponding to the current target task to obtain cleaned multimodal data; The cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data.

[0045] Optionally, after receiving the target task and its corresponding multimodal data, this embodiment cleans the multimodal data to remove noise, errors, and redundant information in the multimodal data to obtain cleaned multimodal data.

[0046] Among them, data cleaning may include but is not limited to missing value processing, outlier detection and processing, duplicate data removal and data format unification. Missing value processing is used to detect and process missing values ​​in the data, such as using interpolation, mean filling or deleting missing samples. Outlier detection and processing is used to detect outliers through statistical methods or machine learning algorithms, and correct or eliminate them. Duplicate data removal is used to identify and delete duplicate multimodal data to avoid interference with the target communication model. Data format unification is used to convert multimodal data from different sources into a unified format.

[0047] Furthermore, the efficiency of data cleaning can be improved through automated data cleaning tools, and the data cleaning effect can be quantified through data quality assessment indicators.

[0048] In addition, the cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data, wherein the preset scale range can be determined according to the characteristics of the multimodal data and the requirements of the target communication model, and the preset scale range can be [0, 1] or [-1, 1].

[0049] Furthermore, the normalization parameters can be dynamically adjusted according to the data distribution, and the multimodal data features that are insensitive to outliers can be normalized through feature scaling technology.

[0050] Optionally, this embodiment improves the performance and efficiency of multimodal data processing tasks and enhances self-evolution stability and generalization capabilities by performing data cleaning and normalization on multimodal data.

[0051] In some embodiments, the target-specific model further includes a fusion mechanism; the target-specific model processes multimodal data features and multimodal data to obtain a target result corresponding to the target task, including: Processing multimodal data features and multimodal data through a target-specific model to obtain specialized features; Determine the general correlation index between the multimodal data features and the current target task, and the special correlation index between the special features and the current target task based on the fusion mechanism; and determine the general weight corresponding to the multimodal data features and the special weight corresponding to the special features based on the general correlation index and the special correlation index; Based on general weights and special weights, multimodal data features and special features are fused to obtain the target results corresponding to the target task.

[0052] Optionally, a fusion mechanism is also present in the target-specific model of this embodiment, wherein the fusion mechanism is used to effectively fuse the multimodal data features obtained from the target general model with the specific features extracted from the target-specific model to generate a more accurate and comprehensive target result.

[0053] Furthermore, the fusion mechanism may include but is not limited to an attention mechanism and a gating mechanism, wherein the attention mechanism can be used to dynamically assign weights of different features, and the gating mechanism can control the fusion ratio.

[0054] In addition, multimodal data features and multimodal data are processed through a target-specific model to generate special features that can reflect the current target task, wherein the target-specific model includes professional domain knowledge corresponding to the target task, and the special features can reflect the characteristics of the current target task.

[0055] In addition, the general correlation index between the multimodal data features and the current target task, as well as the special correlation index between the special features and the current target task are determined through the fusion mechanism, wherein the general correlation index and the special correlation index are used to indicate the importance of the multimodal data features and the special features to the current target task.

[0056] Among them, general correlation indicators and special correlation indicators can be determined through correlation analysis or attention mechanism.

[0057] In addition, based on the general relevance index and the special relevance index, the fusion mechanism assigns general weights to multimodal data features and special weights to special features. The size of the general weight and the size of the special weight reflect the importance of the corresponding features in the target task.

[0058] The general weights and the special weights may be normalized so that the sum of the general weights and the special weights is 1. The general weights and the special weights may also be determined by optimizing the back propagation algorithm through learnable weight parameters.

[0059] In addition, the fusion mechanism performs weighted fusion of multimodal data features and dedicated features according to general weights and dedicated weights to obtain the fused feature representation, and generates the target result corresponding to the target task based on the feature representation.

[0060] This embodiment achieves the accuracy and task adaptability of multimodal data fusion. The fusion mechanism enables multimodal data features and dedicated features to dynamically collaborate to obtain the target result according to the requirements of the target task.

[0061] In some embodiments, updating the general-purpose model parameters of the trained target general-purpose model based on historical multimodal data features, current multimodal data, and target results includes: For the current multimodal data, receiving feedback of the current target result, and correcting the target result based on the feedback to obtain a corrected target result; Determine the loss value between the corrected target result and the target result through a preset loss function, determine the gradient value of the current target general-purpose model based on the loss value, and determine the update direction of the general-purpose model parameters of the target general-purpose model; Receives a preset learning rate and updates the general model parameters based on the optimization algorithm, gradient value, historical multimodal data features, and the preset learning rate.

[0062] Optionally, after obtaining the target result, this embodiment receives feedback results from the outside (such as a doctor's confirmation or correction of the diagnosis result, a user's rating of the target result, etc.), wherein the feedback results are used to correct the target result to obtain a more accurate corrected target result.

[0063] Furthermore, feedback results can be collected in real time through the human-computer interaction interface, or obtained in batches through historical data annotation. During the correction process, they can be corrected through weighted averaging, confidence correction, or expert rule correction. For example, if the confidence difference between the feedback result and the target result is large, a larger correction will be made to the target result of the target general-purpose model; otherwise, a smaller correction will be made.

[0064] In addition, a loss value between the corrected target result and the target result is determined by a preset loss function (such as mean square error, cross entropy loss, contrast loss, etc.), wherein the loss value reflects the gap between the corrected target result and the target result.

[0065] Among them, the preset loss function can select different preset loss functions according to different target tasks. For example, for classification tasks, the cross entropy loss function can be used, for regression tasks, the mean square error loss function can be used, and for multimodal tasks, the multi-task loss function can be selected to perform weighted summation of the outputs of different modalities.

[0066] In addition, the backpropagation algorithm can be used to determine the gradient value of the loss value with respect to the general model parameters of the current target general model. The gradient value represents the rate of change of the preset loss function with respect to the target general model parameters, indicating the optimization direction of the general model parameters.

[0067] Furthermore, the gradient value can be determined through automatic differentiation technology (such as Autograd in PyTorch and TensorFlow), and the gradient explosion problem can be prevented through gradient clipping. For multimodal data, the gradients of different modes can be determined separately and then fused.

[0068] In addition, it receives the preset learning rate and combines it with the optimization algorithm (such as stochastic gradient descent SGD, Adam, RMSprop, etc.) to update the general model parameters of the target general model according to the gradient value, optimization algorithm, preset learning rate and historical multimodal data characteristics.

[0069] Among them, the Adam algorithm is used to adaptively adjust the learning rate and is suitable for most scenarios. The RMSprop algorithm is used to deal with non-stationary targets. The SGD with momentum is used to accelerate convergence and reduce oscillations.

[0070] Furthermore, the learning rate adjustment strategy can be: Learning rate decay: Gradually reduce the learning rate as the number of training rounds increases; Adaptive learning rate: Dynamically adjust the learning rate according to the gradient value.

[0071] Furthermore, historical multimodal data features can be stored in a preset memory library to prevent catastrophic forgetting, and can also be integrated into the current target general model through knowledge distillation.

[0072] This embodiment improves the incremental learning efficiency and knowledge retention capability of the target general-purpose model and reduces the forgetting of old knowledge.

[0073] In some embodiments, receiving a preset learning rate, and updating general-purpose model parameters based on an optimization algorithm, a gradient value, historical multimodal data features, and the preset learning rate, includes: Extract historical multimodal data features from the preset memory library, use the historical multimodal data features as the regularization term of the preset loss function, and update the loss value and gradient value; Receive a preset learning rate, update the general model parameters based on the optimization algorithm, the updated gradient value and the preset learning rate, and update the historical multimodal data features in the preset memory library based on the current multimodal data features.

[0074] Optionally, this embodiment extracts historical multimodal data features from a preset memory bank, wherein the preset memory bank stores important data features encountered by the current target general-purpose model in the past learning process, indicating the old knowledge that the current target general-purpose model has mastered. By introducing historical multimodal data features into the current target general-purpose model, it is possible to effectively prevent the target general-purpose model from forgetting old knowledge when learning new data, thereby achieving the ability of continuous learning.

[0075] Furthermore, the preset memory library can adopt data structures such as a ring buffer, a least recently used (LRU) cache or a feature vector database. During the feature extraction process of historical multimodal data, the current multimodal data features can be screened according to their importance or relevance to the current target task.

[0076] In addition, the historical multimodal data features are used as regularization terms and added to the preset loss function, and the loss value is updated, and the gradient value is updated according to the updated loss value. The regularization term is used to constrain the complexity of the target general-specific model to prevent overfitting. It can also guide the target general-specific model to maintain a balance between new and old knowledge, thereby improving the generalization ability of the target general-specific model.

[0077] Furthermore, the regularization term corresponding to the historical multimodal data features can determine the distance between the target result of the current target general model and the historical multimodal data features in the form of L2 norm. As part of the loss function, the weight of the regularization term can also be dynamically adjusted according to the time distance of the historical multimodal data features or the relevance to the current target task, so that the target general model pays more attention to recent target tasks and key target tasks.

[0078] In addition, after the preset loss function and gradient value are updated, the preset learning rate is received and combined with the optimization algorithm (such as stochastic gradient descent SGD, Adam, RMSprop, etc.) to update the general model parameters of the target general model. Among them, the preset learning rate can be set according to the characteristics of the target task or the dynamic adjustment strategy.

[0079] Specifically, based on the updated loss value, the gradient value of the general model parameters is determined, the gradient is processed through the optimization algorithm (such as momentum acceleration, adaptive learning rate adjustment, etc.), and the general model parameters are updated in combination with the preset learning rate.

[0080] While updating the communication model parameters, the historical multimodal data features in the preset memory library are updated through the current multimodal data features. The current multimodal data features can be integrated into the preset memory library through a replacement strategy (such as least recently used LRU) or a fusion strategy (such as weighted average) to maintain the timeliness and representativeness of the content in the preset memory library.

[0081] Furthermore, the learning rate scheduler (such as Step Decay, Cosine Annealing, Warmup, etc.) can be used to dynamically adjust the learning rate and improve the convergence effect of the target general-purpose model.

[0082] Furthermore, if the current multimodal data is multiple groups of multimodal data, and the amount of multimodal data exceeds a preset quantity threshold, the multimodal data can be divided according to the preset quantity threshold to process the multimodal data in batches, and update the communication model parameters according to each batch of multimodal data.

[0083] This embodiment realizes the continuous learning and stable performance of the target communication model by introducing historical multimodal data features and dynamically updating the preset memory library, combining the optimization algorithm and learning rate scheduling strategy, and is suitable for complex multimodal and multi-task application scenarios.

[0084] In some embodiments, after monitoring the performance indicators of the trained target communication model according to a preset monitoring period, the method further includes: When the performance indicators meet the preset performance indicators, monitor the changing trend of the performance indicators within multiple consecutive preset monitoring cycles; If the performance indicator meets the preset performance indicator within the preset monitoring period of the consecutive preset number of periods, the self-evolution process of the target communication model is stopped, and the communication model parameters of the current target communication model are saved; If the performance indicator does not meet the preset performance indicator within the preset monitoring period of the consecutive preset number of periods, the trained target communication model is adjusted according to the preset adjustment method.

[0085] Optionally, when this embodiment detects that the performance indicators of the target communication model (such as accuracy, recall rate, F1 score, inference delay, etc.) reach or exceed the preset performance indicators within the preset monitoring period, the optimization or self-evolution process of the model will not be stopped immediately, and the performance indicators will be further monitored in the next multiple consecutive preset monitoring periods. Trend monitoring is used to ensure the stability and continuity of the performance of the target communication model, rather than accidental peak performance.

[0086] Specifically, the performance indicators within each preset monitoring period can be recorded to form a performance indicator change curve. By analyzing the performance indicator change curve, it can be determined whether the model performance is continuously stable, gradually improving, or showing a fluctuating or downward trend.

[0087] Furthermore, the trend of change can be determined through a sliding window mechanism. That is, a sliding window of fixed length is set (for example, the last five monitoring cycles) and statistical analysis is performed on the performance indicators within the window, such as calculating the mean, variance, slope, etc., to determine the stability and trend of the performance.

[0088] The trend of change can also be determined through trend detection algorithms, that is, time series analysis methods such as linear regression, exponential smoothing, and moving average can be used to detect the trend of change in performance indicators.

[0089] It is also possible to determine the trend of change through abnormal fluctuation detection, that is, to introduce an anomaly detection mechanism to identify abnormal fluctuations in performance indicators and avoid misjudgment.

[0090] Furthermore, if the target communication model's performance indicators meet the preset performance indicators over multiple consecutive preset monitoring periods (e.g., five consecutive periods), and the performance trend is stable or continuously improving, the target communication model can be considered to have achieved the expected performance standards and possess sufficient stability. At this point, the system will automatically stop the model's self-evolution process (such as parameter updates and structural adjustments) and save the current target communication model parameters.

[0091] At the same time, the performance indicators, training logs, parameter configurations and other information of the target communication model can be saved together for subsequent auditing, analysis and reproduction.

[0092] Furthermore, the target general-purpose model can be versioned, that is, model version control tools (such as MLflow and DVC) can be used to manage the general-purpose model parameters and record each saved model version and its corresponding performance indicators.

[0093] You can also set up an automatic save mechanism, that is, set an automatic save strategy, for example, automatically trigger the save operation after the performance indicators are stable and reach a certain preset monitoring period, avoiding manual intervention.

[0094] Furthermore, when saving general-purpose model parameters, the target general-purpose model can be optimized through model compression techniques (such as quantization and pruning) to reduce storage space and inference latency.

[0095] In addition, if it is found that the performance indicators of the target communication and special model fail to continue to meet the preset performance indicators during multiple consecutive preset monitoring cycles, or the performance shows obvious fluctuations or even a downward trend, it is considered that the target communication and special model has not yet reached a stable state and needs further optimization. The preset adjustment method of the target communication and special model is automatically triggered to adjust the target communication and special model.

[0096] Among them, the preset adjustment method includes structural adjustment of the target general-specific model and / or adjustment of the update frequency corresponding to the general-specific model parameters. Among them, the model structure adjustment can be to increase or decrease the number of layers of the target general-specific model, adjust the number of heads of the attention mechanism, and modify the network connection method to improve the expressive ability or generalization ability of the target general-specific model.

[0097] Adjusting the update frequency corresponding to the general-purpose model parameters may include, but is not limited to, adjusting the learning rate, optimizer type, and batch size to improve the training effect of the target general-purpose model.

[0098] Furthermore, the preset adjustment method can also include data enhancement strategy adjustment and regularization strategy adjustment, wherein the data enhancement strategy adjustment is to introduce a new data enhancement method or adjust the existing enhancement strategy to improve the generalization ability of the target general-purpose model, and the regularization strategy adjustment is to adjust the Dropout ratio, add a new regularization term, for example, to prevent the target general-purpose model from overfitting.

[0099] Furthermore, the target communication model can be adjusted in the following ways: Automatic hyperparameter optimization, that is, automatically searching for the optimal hyperparameter combination through Bayesian optimization, genetic algorithms, reinforcement learning, etc., neural architecture search (NAS), that is, automatically searching for the optimal model structure to improve model performance, and online learning strategy adjustment, that is, dynamically adjusting parameters such as learning rate and batch size based on performance indicator feedback to achieve adaptive optimization of the target general-purpose model.

[0100] This embodiment realizes the automatic determination of the target communication model optimization termination timing by continuously monitoring the performance indicators of the target communication model, and automatically adjusts the target communication model when necessary, thereby ensuring the stability and reliability of the performance indicators of the target communication model.

[0101] In order to effectively solve the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improve the synergy effect of the multimodal general-specialty model and the update effect of updating according to new knowledge, the present application provides an embodiment of a multimodal general-specialty model self-evolution system based on incremental learning for realizing all or part of the content of the multimodal general-specialty model self-evolution based on incremental learning, see Figure 2 The multimodal communication model self-evolution system based on incremental learning specifically includes the following contents: The first processing module 10 is configured to receive a current target task and multimodal data corresponding to the current target task, and determine, based on the target task, a trained target-specific model corresponding to the target task, wherein the target-specific model includes a target-general model and a target-specific model; The second processing module 20 is used to extract data features of the multimodal data using the target general model to obtain multimodal data features, and process the multimodal data features and the multimodal data using the target specific model to obtain a target result corresponding to the target task; An updating module 30 is configured to obtain historical multimodal data features corresponding to the historical multimodal data, and update general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data, and the target result; The adjustment module 40 is used to monitor the performance indicators of the trained target communication model according to a preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, the trained target communication model is adjusted according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0102] From the above description, it can be seen that the embodiment of the present application provides a multimodal communication model self-evolution system based on incremental learning, which can innovatively receive the current target task and the multimodal data corresponding to the current target task, determine the trained target communication model corresponding to the target task according to the target task, extract the data features of the multimodal data through the target general model in the target communication model, obtain the multimodal data features, generate the target result through the target-specific model in the target communication model combining the multimodal data features and the multimodal data, and update the communication model of the target communication model according to the historical multimodal data features, the current multimodal data and the target result. Model parameters, monitor the performance indicators of the trained target general-purpose model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target general-purpose model according to the preset adjustment method. It can dynamically adjust the structure and parameters of the target general-purpose model through incremental learning and collaborative updating of the target general-purpose model, reducing the need for full retraining, while improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-purpose model's ability to extract features from multimodal data and adapt to target tasks, and improving the accuracy and robustness of the target results. This method effectively solves the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improves the synergistic effect of multimodal general-purpose models and the update effect based on new knowledge.

[0103] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and to significantly improve the synergy of multimodal general-specific models and the update effect based on new knowledge, the present application provides an embodiment of an electronic device for implementing all or part of the content of the multimodal general-specific model self-evolution method based on incremental learning. The electronic device specifically includes the following content: Processor, memory, communication interface, and bus; wherein the processor, memory, and communication interface communicate with each other via the bus; the communication interface is used to implement information transmission between a multimodal communication model self-evolution system based on incremental learning and related devices such as core business systems, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited to this. In this embodiment, the logic controller can be implemented with reference to the embodiment of a multimodal communication model self-evolution method based on incremental learning and the embodiment of a multimodal communication model self-evolution system based on incremental learning in the embodiment, and their contents are merged here, and repeated parts are not repeated.

[0104] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0105] In practical applications, part of the method for self-evolution of a multimodal communication model based on incremental learning can be executed on the electronic device side as described above, or all operations can be completed on the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed on the client device, the client device may also include a processor.

[0106] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0107] Figure 3 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0108] In one embodiment, a multimodal communication model self-evolution method based on incremental learning can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: Step S101: receiving a current target task and multimodal data corresponding to the current target task, and determining a trained target-specific model corresponding to the target task based on the target task, wherein the target-specific model includes a target-general model and a target-specific model; Step S102: extracting data features of the multimodal data through the target general model to obtain multimodal data features, and processing the multimodal data features and the multimodal data through the target specific model to obtain a target result corresponding to the target task; Step S103: Acquire historical multimodal data features corresponding to the historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data, and the target result; Step S104: Monitor the performance indicators of the trained target communication model according to a preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target communication model according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0109] As can be seen from the above description, the electronic device provided in the embodiment of the present application innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target general-specific model corresponding to the target task according to the target task, extracts data features of the multimodal data through the target general model in the target general-specific model, obtains multimodal data features, generates a target result by combining the multimodal data features and the multimodal data through the target-specific model in the target general-specific model, updates the general-specific model parameters of the target general-specific model according to the historical multimodal data features, the current multimodal data and the target result, monitors the performance indicators of the trained target general-specific model according to the preset monitoring period, and if the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjusts the trained target general-specific model according to the preset adjustment method, and can dynamically adjust the structure and parameters of the target general-specific model through incremental learning and collaborative updating of the target general-specific model, reducing the need for full retraining, while improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-specific model's ability to extract features from multimodal data and adapt to target tasks, and improving the accuracy and robustness of the target result. This method effectively solves the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improves the synergy effect of multimodal general-specific models and the update effect based on new knowledge.

[0110] In another embodiment, a multimodal communication model self-evolution system based on incremental learning can be configured separately from the central processing unit 9100. For example, a multimodal communication model self-evolution system based on incremental learning can be configured as a chip connected to the central processing unit 9100, and the function of a multimodal communication model self-evolution method based on incremental learning can be realized through the control of the central processing unit.

[0111] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 3In addition, the electronic device 9600 may also include all components shown in Figure 3 For components not shown, reference may be made to the prior art.

[0112] like Figure 3 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0113] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0114] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0115] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0116] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0117] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0118] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless local area network modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130, providing audio output via the speaker 9131 and receiving audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0119] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the method for self-evolution of a multimodal communication model based on incremental learning in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program implements all steps of the method for self-evolution of a multimodal communication model based on incremental learning in the above-mentioned embodiment, where the execution subject is a server or a client. For example, when the processor executes the computer program, the following steps are implemented: Step S101: receiving a current target task and multimodal data corresponding to the current target task, and determining a trained target-specific model corresponding to the target task based on the target task, wherein the target-specific model includes a target-general model and a target-specific model; Step S102: extracting data features of the multimodal data through the target general model to obtain multimodal data features, and processing the multimodal data features and the multimodal data through the target specific model to obtain a target result corresponding to the target task; Step S103: Acquire historical multimodal data features corresponding to the historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data, and the target result; Step S104: Monitor the performance indicators of the trained target communication model according to a preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target communication model according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0120] As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target general-specific model corresponding to the target task according to the target task, extracts data features of the multimodal data through the target general model in the target general-specific model, obtains multimodal data features, generates a target result by combining the multimodal data features and the multimodal data through the target-specific model in the target general-specific model, updates the general-specific model parameters of the target general-specific model according to the historical multimodal data features, the current multimodal data, and the target result, monitors the performance indicators of the trained target general-specific model according to a preset monitoring period, and adjusts the trained target general-specific model according to a preset adjustment method if the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators. The structure and parameters of the target general-specific model can be dynamically adjusted through incremental learning and collaborative updating of the target general-specific model, reducing the need for full retraining, while improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-specific model's ability to extract features from multimodal data and adapt to target tasks, and improving the accuracy and robustness of the target result. This method effectively solves the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improves the synergy effect of multimodal general-specific models and the update effect based on new knowledge.

[0121] Embodiments of the present application also provide a computer program product capable of implementing all steps of the method for self-evolution of a multimodal communication model based on incremental learning in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instructions are executed by a processor, the computer program / instructions implement the steps of the method for self-evolution of a multimodal communication model based on incremental learning. For example, the computer program / instructions implement the following steps: Step S101: receiving a current target task and multimodal data corresponding to the current target task, and determining a trained target-specific model corresponding to the target task based on the target task, wherein the target-specific model includes a target-general model and a target-specific model; Step S102: extracting data features of the multimodal data through the target general model to obtain multimodal data features, and processing the multimodal data features and the multimodal data through the target specific model to obtain a target result corresponding to the target task; Step S103: Acquire historical multimodal data features corresponding to the historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data, and the target result; Step S104: Monitor the performance indicators of the trained target communication model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target communication model according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

[0122] As can be seen from the above description, the computer program product provided in the embodiment of the present application innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target general-specific model corresponding to the target task according to the target task, extracts data features of the multimodal data through the target general model in the target general-specific model, obtains multimodal data features, generates a target result by combining the multimodal data features and the multimodal data through the target-specific model in the target general-specific model, updates the general-specific model parameters of the target general-specific model according to the historical multimodal data features, the current multimodal data, and the target result, monitors the performance indicators of the trained target general-specific model according to a preset monitoring period, and if the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjusts the trained target general-specific model according to a preset adjustment method. The structure and parameters of the target general-specific model can be dynamically adjusted through incremental learning and collaborative updating of the target general-specific model, reducing the need for full retraining, while improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-specific model's ability to extract features from multimodal data and adapt to target tasks, and improving the accuracy and robustness of the target result. This method effectively solves the shortcomings of traditional technologies in terms of insufficient synergy between general knowledge and specialized knowledge, and significantly improves the synergy effect of multimodal general-specific models and the update effect based on new knowledge.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal communication model self-evolution method based on incremental learning, characterized in that: The method comprises: Receive a current target task and multimodal data corresponding to the current target task, and determine, based on the target task, to execute a trained target-specific model corresponding to the target task, wherein the target-specific model includes a target-general model and a target-specific model; Extracting data features of the multimodal data through the target general model to obtain multimodal data features, and processing the multimodal data features and the multimodal data through the target specific model to obtain a target result corresponding to the target task; Obtain historical multimodal data features corresponding to the historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data, and the target result; Monitor the performance indicators of the trained target communication model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target communication model according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

2. The method according to claim 1, characterized in that After receiving the current target task and the multimodal data corresponding to the current target task, the method further includes: Performing data cleaning on the multimodal data corresponding to the current target task to obtain cleaned multimodal data; The cleaned multimodal data is mapped to a preset scale range to obtain the normalized multimodal data.

3. The method according to claim 1, characterized in that The target-specific model also includes a fusion mechanism; the target-specific model processes the multimodal data features and the multimodal data to obtain the target result corresponding to the target task, including: Processing the multimodal data features and the multimodal data through the target-specific model to obtain a specific feature; Determining, based on the fusion mechanism, a general relevance index between the multimodal data feature and the current target task, and a specific relevance index between the specific feature and the current target task, and determining, based on the general relevance index and the specific relevance index, a general weight corresponding to the multimodal data feature and a specific weight corresponding to the specific feature; Based on the general weight and the special weight, the multimodal data features and the special features are fused to obtain the target result corresponding to the target task.

4. The method according to claim 1, wherein The updating of the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, the current multimodal data and the target result includes: For the current multimodal data, receiving a feedback result of the current target result, and correcting the target result based on the feedback result to obtain the corrected target result; Determine the loss value between the corrected target result and the target result by using a preset loss function, and determine the gradient value of the current target general-purpose model based on the loss value to determine the update direction of the general-purpose model parameters of the target general-purpose model; A preset learning rate is received, and the general model parameters are updated based on the optimization algorithm, the gradient value, the historical multimodal data features, and the preset learning rate.

5. The method according to claim 4, characterized in that The receiving a preset learning rate and updating the general-purpose model parameters based on the optimization algorithm, the gradient value, the historical multimodal data features, and the preset learning rate include: Extracting the historical multimodal data features from a preset memory library, using the historical multimodal data features as regularization terms of the preset loss function, and updating the loss value and the gradient value; A preset learning rate is received, and based on the optimization algorithm, the updated gradient value and the preset learning rate, the general model parameters are updated, and the historical multimodal data features in the preset memory bank are updated based on the current multimodal data features.

6. The method according to claim 1, characterized in that After monitoring the performance indicators of the trained target communication model according to the preset monitoring period, the method further includes: When the performance indicator satisfies the preset performance indicator, monitoring the change trend of the performance indicator within a plurality of consecutive preset monitoring periods; If the performance indicator satisfies the preset performance indicator within the preset monitoring period for a consecutive preset number of periods, stopping the self-evolution process of the target communication-specific model and saving the communication-specific model parameters of the current target communication-specific model; If the performance indicator does not meet the preset performance indicator within the preset monitoring period for a continuous preset number of periods, the trained target communication model is adjusted according to a preset adjustment method.

7. A multimodal communication model self-evolution system based on incremental learning, characterized by: The system comprises: A first processing module is configured to receive a current target task and multimodal data corresponding to the current target task, and determine, based on the target task, to execute a trained target-specific model corresponding to the target task, wherein the target-specific model includes a target-general model and a target-specific model; A second processing module is configured to extract data features of the multimodal data using the target general model to obtain multimodal data features, and process the multimodal data features and the multimodal data using the target specific model to obtain a target result corresponding to the target task; An update module is used to obtain historical multimodal data features corresponding to historical multimodal data, and update the general-purpose model parameters of the trained target general-purpose model based on the historical multimodal data features, current multimodal data and target results; An adjustment module is used to monitor the performance indicators of the trained target communication model according to a preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, the trained target communication model is adjusted according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target communication model and / or adjusting the update frequency corresponding to the communication model parameters.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multimodal communication model self-evolution method based on incremental learning as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for self-evolution of a multimodal communication model based on incremental learning as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for self-evolution of a multimodal communication model based on incremental learning as described in any one of claims 1 to 6 are implemented.

Images