Multimodal communication model self-evolution method and system based on incremental learning

Abstract

This application provides a method and system for the self-evolution of a multimodal general-specific model based on incremental learning. The method involves receiving the current target task and its corresponding multimodal data to determine the corresponding trained target general-specific model; extracting data features from the multimodal data; processing the multimodal data features and multimodal data using a target-specific model to obtain the target result; acquiring historical multimodal data features corresponding to historical multimodal data and updating the parameters of the trained target general-specific model; monitoring the performance indicators of the trained target general-specific model according to a preset monitoring period; and adjusting the trained target general-specific model according to a preset adjustment method if the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators. This method effectively addresses the shortcomings in the synergy between general and specific knowledge, significantly improving the synergy effect and the update effect based on new knowledge.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method and system for the self-evolution of multimodal general-purpose models based on incremental learning. Background Technology

[0002] Current processing of multimodal data mainly relies on static model architecture. When a new task is received, the static model needs to be retrained with the full amount of data, which results in high consumption of computing resources and the static model may suffer from catastrophic forgetting.

[0003] In existing technologies, in order to solve the problems of high computational resource consumption and potential catastrophic forgetting, optimization can be achieved by training a hybrid model with a fixed architecture. However, the hybrid model with a fixed architecture cannot dynamically adjust the synergistic relationship between the general model and the special model according to the target task, and fails to achieve autonomous synergistic optimization of model structure and parameters. It is difficult to take into account both general and professional needs, and the cost of retraining the model is high. The retrained model is also prone to forgetting old knowledge. Summary of the Invention

[0004] To address the problems in existing technologies, this application provides a method and system for the self-evolution of multimodal general-specific models based on incremental learning. This method effectively solves the shortcomings of traditional technologies in terms of insufficient synergy between general and specific knowledge, and significantly improves the synergistic effect of multimodal general-specific models and the updating effect based on new knowledge.

[0005] To solve at least one of the above problems, this application provides the following technical solution:

[0006] Firstly, this application provides a self-evolution method for multimodal general-purpose models based on incremental learning, including:

[0007] Receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general and target specific models to be executed based on the target task. The target general and target specific models include a target general model and a target specific model.

[0008] Data features of multimodal data are extracted by using a general target model to obtain multimodal data features. Multimodal data features and multimodal data are then processed by a target-specific model to obtain the target result corresponding to the target task.

[0009] Obtain the historical multimodal data features corresponding to the historical multimodal data, and update the parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result;

[0010] The performance metrics of the trained target general-purpose model are monitored according to the preset monitoring cycle. If the performance metrics corresponding to the current preset monitoring cycle do not meet the preset performance metrics, the trained target general-purpose model is adjusted according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

[0011] Furthermore, after receiving the current target task and the corresponding multimodal data, it also includes:

[0012] Data cleaning is performed on the multimodal data corresponding to the current target task to obtain cleaned multimodal data;

[0013] The cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data.

[0014] Furthermore, it also includes: processing multimodal data features and multimodal data through a target-specific model to obtain specific features;

[0015] Based on the fusion mechanism, the general correlation index and the specific correlation index of the current target task are determined for the multimodal data features. Based on the general correlation index and the specific correlation index, the general weight and the specific weight corresponding to the multimodal data features are determined.

[0016] Based on general weights and specific weights, multimodal data features and specific features are integrated to obtain the target results corresponding to the target task.

[0017] Furthermore, it also includes: for the current multimodal data, receiving feedback results of the current target result, correcting the target result based on the feedback results, and obtaining the corrected target result;

[0018] The loss value between the corrected target result and the target result is determined by a preset loss function. Based on the loss value, the gradient value of the current target general-purpose model is determined, so as to determine the update direction of the general-purpose model parameters of the target general-purpose model.

[0019] Receive the preset learning rate and update the general-purpose model parameters based on the optimization algorithm, gradient values, historical multimodal data features, and the preset learning rate.

[0020] Furthermore, it also includes: extracting historical multimodal data features from a preset memory, using the historical multimodal data features as a regularization term of a preset loss function, and updating the loss value and gradient value;

[0021] Receive the preset learning rate, update the general model parameters based on the optimization algorithm, the updated gradient values ​​and the preset learning rate, and update the historical multimodal data features in the preset memory based on the current multimodal data features.

[0022] Furthermore, after monitoring the performance metrics of the trained target general-purpose model according to a preset monitoring cycle, the process also includes:

[0023] Under the condition that the performance indicators meet the preset performance indicators, monitor the changing trend of the performance indicators over multiple consecutive preset monitoring periods;

[0024] If the performance indicators meet the preset performance indicators within a preset number of consecutive monitoring periods, stop the self-evolution process of the target general-purpose model and save the general-purpose model parameters of the current target general-purpose model.

[0025] If the performance indicators do not meet the preset performance indicators within the preset monitoring period of consecutive preset periods, the trained target general-purpose model will be adjusted according to the preset adjustment method.

[0026] Secondly, this application provides a self-evolving system for a multimodal general-purpose model based on incremental learning, comprising:

[0027] The first processing module is used to receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general-purpose model corresponding to the target task based on the target task. The target general-purpose model includes a target general model and a target specific model.

[0028] The second processing module is used to extract data features of multimodal data through the target general model to obtain multimodal data features, and to process the multimodal data features and multimodal data through the target specific model to obtain the target result corresponding to the target task.

[0029] The update module is used to obtain the historical multimodal data features corresponding to the historical multimodal data, and update the parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result.

[0030] The adjustment module is used to monitor the performance indicators of the trained target general-purpose model according to a preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, the trained target general-purpose model is adjusted according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

[0031] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the self-evolution method for a multimodal communication model based on incremental learning.

[0032] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the self-evolution method for a multimodal communication model based on incremental learning.

[0033] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the self-evolution method for a multimodal communication model based on incremental learning.

[0034] As can be seen from the above technical solution, this application provides a method and system for the self-evolution of a multimodal general-specific model based on incremental learning. It innovatively receives the current target task and the corresponding multimodal data, determines the trained target general-specific model corresponding to the target task based on the target task, extracts data features from the multimodal data through the target general model in the target general-specific model, obtains multimodal data features, generates the target result by combining the multimodal data features and multimodal data through the target specific model in the target general-specific model, and updates the target general-specific model based on historical multimodal data features, current multimodal data, and the target result. The method monitors the performance metrics of the trained target general-specific model according to a preset monitoring period. If the performance metrics corresponding to the current preset monitoring period do not meet the preset performance metrics, the trained target general-specific model is adjusted according to a preset adjustment method. This allows for dynamic adjustment of the target general-specific model's structure and parameters through incremental learning and collaborative updates, reducing the need for full retraining. It also improves the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-specific model's ability to extract features from multimodal data and adapt to target tasks, improving the accuracy and robustness of the target results. This method effectively addresses the shortcomings of traditional techniques in terms of insufficient synergy between general and specialized knowledge, significantly improving the synergistic effect of multimodal general-specific models and the update effect based on new knowledge. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a self-evolution method for a multimodal general-purpose model based on incremental learning, as described in an embodiment of this application.

[0037] Figure 2This is a structural diagram of a self-evolving system for a multimodal general-purpose model based on incremental learning, as described in an embodiment of this application.

[0038] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0039] Figure label:

[0040] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0043] In current technologies, with the development of artificial intelligence, the demand for processing multimodal data (such as text, image, and audio data) is increasing. Traditional multimodal models often require retraining when faced with new multimodal data and target tasks, which is not only time-consuming and labor-intensive but also prone to problems such as forgetting old knowledge. Incremental learning allows models to continuously absorb new data and knowledge based on existing knowledge, achieving gradual updates and optimization. However, the application of incremental learning in multimodal models is not yet perfect, lacking a system and method that can effectively coordinate and self-evolve between general and specialized knowledge to adapt to the complex and ever-changing demands of multimodal data processing.

[0044] To effectively address the shortcomings of traditional technologies in terms of insufficient synergy between general and specialized knowledge, and to significantly improve the synergistic effect and update efficiency of multimodal general-specific models based on new knowledge, this application provides an embodiment of a self-evolution method for multimodal general-specific models based on incremental learning. See [link to embodiment]. Figure 1 The self-evolution method for multimodal general knowledge models based on incremental learning specifically includes the following:

[0045] Step S101: Receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general-purpose model corresponding to the target task based on the target task.

[0046] The target general model includes a target general model and a target specific model.

[0047] Optionally, this embodiment receives the current target task and its multimodal data, wherein the multimodal data includes at least two of image data, text data, and audio data. The target task can be analyzed to determine its task type and characteristics, thereby determining the trained target-specific model corresponding to the target task. The target-specific model includes a target-specific model and a target-general model.

[0048] Furthermore, the target task can be encoded by a task encoder to generate a task feature vector, which can then be used to determine the task general model for executing the target task.

[0049] This embodiment realizes the determination of target general models that can perform target tasks through target tasks, thereby enabling accurate processing of multimodal data corresponding to target tasks.

[0050] Step S102: Extract data features of multimodal data through the target general model to obtain multimodal data features, process the multimodal data features and multimodal data through the target specific model to obtain the target result corresponding to the target task.

[0051] Optionally, in this embodiment, data features of multimodal data are extracted through a target general model to obtain multimodal data features, wherein the multimodal data features include image features corresponding to image data, text features corresponding to text data, and audio features corresponding to audio data.

[0052] Different feature extraction methods can be adopted according to the different data types of multimodal data. For example, image features of image data can be extracted by convolutional neural networks. The convolutional and pooling layers of the convolutional neural network can learn features such as edges, textures, and shapes in the image data to generate image features.

[0053] Text features can be extracted from text data by using recurrent neural networks or transformer models. Recurrent neural networks can capture the sequence information in text data and obtain text features based on the sequence information. Transformer models can process text in parallel through self-attention mechanisms to generate semantic feature vectors (i.e., text features).

[0054] Audio features are obtained by extracting audio data using an audio feature extraction algorithm.

[0055] In addition, by using a target-specific model to process multimodal data features and multimodal data, it is possible to combine the professional domain knowledge and rules corresponding to the target task to conduct refined analysis of multimodal data and obtain the target results corresponding to the target task.

[0056] This embodiment achieves greater semantic understanding depth and domain adaptability for multimodal data, improves the collaborative efficiency of multimodal data features obtained through a general task model and target results obtained through a task-specific model, and enhances the accuracy and robustness of target result generation.

[0057] Step S103: Obtain the historical multimodal data features corresponding to the historical multimodal data, and update the general-specific model parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result.

[0058] Optionally, this embodiment obtains historical multimodal data features corresponding to historical multimodal data. The historical multimodal data features are obtained by processing the historical multimodal data through the target general model and can be stored in a preset memory.

[0059] The parameters of the trained target general-specific model are updated based on historical multimodal data features, current multimodal data, and target results. The joint loss function can be determined by combining the target results. The joint loss function includes a fitting term for the current multimodal data features and a retention constraint term for the historical multimodal data features, thereby optimizing the general-specific model parameters.

[0060] This embodiment achieves the ability to retain historical knowledge during incremental learning, prevents catastrophic forgetting, improves the model's adaptability to dynamic multimodal environments, enhances the accuracy of general-purpose model parameter updates, and enables the target general-purpose model to continuously meet the performance requirements of the target results.

[0061] Step S104: Monitor the performance indicators of the trained target general-purpose model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target general-purpose model according to the preset adjustment method.

[0062] The preset adjustment methods include adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the general-purpose model parameters.

[0063] Optionally, this embodiment evaluates the performance of the trained target general-purpose model according to a preset monitoring period (e.g., weekly, monthly). The performance indicators include accuracy, recall, F1 score, inference time, etc. If the performance indicators in the current monitoring period do not reach the preset performance indicators, the adjustment mechanism will be automatically triggered.

[0064] In addition, when it is detected that the performance index corresponding to the current preset monitoring period has not reached the preset performance index threshold, the trained target general-purpose model is adjusted according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the general-purpose model parameters.

[0065] Adjusting the model structure involves modifying the target general module and / or target professional model of the trained target general-purpose model, such as by increasing or decreasing the number of network layers or optimizing the design of the fusion mechanism to adapt to the new data distribution. Adjusting the parameter update frequency includes, but is not limited to, changing the learning rate of the optimization algorithm, adjusting the gradient calculation batch size, or introducing a knowledge retention mechanism to control the parameter update step size, ensuring that the retention of old knowledge is balanced while absorbing new knowledge. The adjustment process is dynamically executed based on the evaluation results until the performance indicators meet the preset requirements or the evolution stopping condition is reached.

[0066] Furthermore, performance metrics can be calculated by evaluating the performance of the target generalized model on the validation dataset.

[0067] This embodiment realizes the adaptive capability and robustness of the target general-specific model in incremental learning, improves the performance and efficiency of multimodal data processing tasks, and enhances self-evolutionary stability and generalization ability.

[0068] This embodiment enables the target general-purpose model to share knowledge between the task-general model and the task-specific model, thereby improving its generalization ability. It can continuously optimize itself based on new multimodal data, adapt to environmental changes, and ensure the long-term stable operation of the target general-purpose model through performance monitoring and dynamic adjustment.

[0069] In some embodiments, after receiving the current target task and the multimodal data corresponding to the current target task, the method further includes:

[0070] Data cleaning is performed on the multimodal data corresponding to the current target task to obtain cleaned multimodal data;

[0071] The cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data.

[0072] Optionally, after receiving the target task and its corresponding multimodal data, this embodiment cleans the multimodal data to remove noise, errors, and redundant information, thus obtaining cleaned multimodal data.

[0073] Data cleaning can include, but is not limited to, missing value handling, outlier detection and handling, duplicate data removal, and data format standardization. Missing value handling is used to detect and process missing values ​​in the data, such as using interpolation, mean filling, or deleting missing samples. Outlier detection and handling is used to detect outliers through statistical methods or machine learning algorithms and correct or remove them. Duplicate data removal is used to identify and delete duplicate multimodal data to avoid interfering with the target general-purpose model. Data format standardization is used to convert multimodal data from different sources into a unified format.

[0074] Furthermore, the efficiency of data cleaning can be improved through automated data cleaning tools, and the effectiveness of data cleaning can be quantified through data quality assessment indicators.

[0075] In addition, the cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data. The preset scale range can be determined according to the characteristics of the multimodal data and the requirements of the target general model. The preset scale range can be [0,1] or [-1,1].

[0076] Furthermore, the normalization parameters can be dynamically adjusted according to the data distribution, or feature scaling techniques can be used to normalize multimodal data features that are not sensitive to outliers.

[0077] Optionally, this embodiment improves the performance and efficiency of multimodal data processing tasks by cleaning and normalizing the multimodal data, and enhances self-evolutionary stability and generalization ability.

[0078] In some embodiments, the target-specific model further includes a fusion mechanism; by processing multimodal data features and multimodal data through the target-specific model, the target result corresponding to the target task is obtained, including:

[0079] Target-specific models are used to process multimodal data features and multimodal data to obtain specific features;

[0080] Based on the fusion mechanism, the general correlation index and the specific correlation index of the current target task are determined for the multimodal data features. Based on the general correlation index and the specific correlation index, the general weight and the specific weight corresponding to the multimodal data features are determined.

[0081] Based on general weights and specific weights, multimodal data features and specific features are integrated to obtain the target results corresponding to the target task.

[0082] Optionally, the target-specific model in this embodiment also includes a fusion mechanism, which is used to effectively fuse the multimodal data features obtained from the target general model with the specific features extracted from the target-specific model to generate more accurate and comprehensive target results.

[0083] Furthermore, the fusion mechanism may include, but is not limited to, attention mechanisms and gating mechanisms. The attention mechanism can be used to dynamically allocate the weights of different features, and the gating mechanism can control the fusion ratio.

[0084] In addition, by processing multimodal data features and multimodal data through a target-specific model, specific features that can reflect the current target task are generated. The target-specific model includes professional domain knowledge corresponding to the target task, and the specific features can reflect the characteristics of the current target task.

[0085] Furthermore, a fusion mechanism is used to determine the general correlation index between multimodal data features and the current target task, as well as the specific correlation index between specific features and the current target task. The general correlation index and the specific correlation index are used to represent the importance of multimodal data features and specific features to the current target task.

[0086] Among these, general correlation indicators and specific correlation indicators can be determined through correlation analysis or attention mechanisms.

[0087] Furthermore, based on general and specific correlation indicators, the fusion mechanism assigns general weights to multimodal data features and specific weights to specific features. The magnitudes of the general and specific weights reflect the importance of the corresponding features in the target task.

[0088] Among these methods, the general weights and specific weights can be normalized so that the sum of the general weights and specific weights is 1. Alternatively, the general weights and specific weights can be determined by optimizing the backpropagation algorithm using learnable weight parameters.

[0089] In addition, the fusion mechanism performs weighted fusion of multimodal data features and special features based on general weights and special weights to obtain the fused feature representation, and generates the target result corresponding to the target task based on the feature representation.

[0090] This embodiment achieves high accuracy and task adaptability in multimodal data fusion. The fusion mechanism enables multimodal data features and specialized features to dynamically coordinate and obtain the target result according to the needs of the target task.

[0091] In some embodiments, updating the general-purpose model parameters of the trained target general-purpose model based on historical multimodal data features, current multimodal data, and target results includes:

[0092] For the current multimodal data, receive the feedback result of the current target result, correct the target result based on the feedback result, and obtain the corrected target result;

[0093] The loss value between the corrected target result and the target result is determined by a preset loss function. Based on the loss value, the gradient value of the current target general-purpose model is determined, so as to determine the update direction of the general-purpose model parameters of the target general-purpose model.

[0094] Receive the preset learning rate and update the general-purpose model parameters based on the optimization algorithm, gradient values, historical multimodal data features, and the preset learning rate.

[0095] Optionally, after obtaining the target result, this embodiment receives feedback results from external sources (such as confirmation or correction of the diagnosis by a doctor, rating of the target result by a user, etc.). The feedback results are used to correct the target result to obtain a more accurate corrected target result.

[0096] Furthermore, feedback results can be collected in real time through a human-computer interaction interface or obtained in batches through historical data annotation. During the correction process, corrections can be made through weighted averaging, confidence correction, or expert rule correction. For example, if the confidence difference between the feedback result and the target result is large, the target result of the target general-to-specialty model will be corrected more significantly; otherwise, a smaller correction will be made.

[0097] In addition, the loss value between the corrected target result and the target result is determined by a preset loss function (such as mean squared error, cross-entropy loss, contrast loss, etc.). The loss value reflects the gap between the corrected target result and the target result.

[0098] The preset loss function can be selected according to different target tasks. For example, for classification tasks, the cross-entropy loss function can be used; for regression tasks, the mean squared error loss function can be used; and for multimodal tasks, a multi-task loss function can be selected to perform weighted summation of the outputs of different modalities.

[0099] Furthermore, the gradient value of the loss value with respect to the parameters of the target general-purpose model can be determined through the backpropagation algorithm. The gradient value represents the rate of change of the preset loss function with respect to the parameters of the target general-purpose model, indicating the direction of optimization of the general-purpose model parameters.

[0100] Furthermore, gradient values ​​can be determined using automatic differentiation techniques (such as Autograd in PyTorch and TensorFlow), and gradient clipping can be used to prevent gradient explosion. For multimodal data, the gradients of different modalities can be determined separately and then fused.

[0101] In addition, the system receives a preset learning rate and, in conjunction with optimization algorithms (such as stochastic gradient descent SGD, Adam, RMSprop, etc.), updates the general-purpose model parameters of the target general-purpose model based on the gradient value, optimization algorithm, preset learning rate, and historical multimodal data characteristics.

[0102] Among them, the Adam algorithm is used to adaptively adjust the learning rate and is suitable for most scenarios, the RMSprop algorithm is used to handle non-stationary objectives, and the SGD with momentum is used to accelerate convergence and reduce oscillations.

[0103] Furthermore, the learning rate adjustment strategy can be:

[0104] Learning rate decay: The learning rate is gradually reduced as the number of training epochs increases;

[0105] Adaptive learning rate: The learning rate is dynamically adjusted based on the gradient value.

[0106] Furthermore, historical multimodal data features can be stored in a pre-set memory to prevent catastrophic forgetting, and can also be integrated into the current target general-purpose model through knowledge distillation.

[0107] This embodiment improves the incremental learning efficiency and knowledge retention capability of the target general-purpose model, and reduces the forgetting of old knowledge.

[0108] In some embodiments, receiving a preset learning rate and updating the general model parameters based on the optimization algorithm, gradient values, historical multimodal data features, and the preset learning rate includes:

[0109] Extract historical multimodal data features from a pre-defined memory, use these features as a regularization term in a pre-defined loss function, and update the loss and gradient values.

[0110] Receive the preset learning rate, update the general model parameters based on the optimization algorithm, the updated gradient values ​​and the preset learning rate, and update the historical multimodal data features in the preset memory based on the current multimodal data features.

[0111] Optionally, this embodiment extracts historical multimodal data features from a preset memory. The preset memory stores important data features encountered by the current target generalist model during past learning processes, representing the old knowledge that the current target generalist model has mastered. By introducing historical multimodal data features into the current target generalist model, it is possible to effectively prevent the target generalist model from forgetting old knowledge when learning new data, thereby achieving the ability to continuously learn.

[0112] Furthermore, the preset memory can adopt data structures such as a circular buffer, a least recently used (LRU) cache, or a feature vector database. During the historical multimodal data feature extraction process, it can be filtered according to the importance of the current multimodal data features or their relevance to the current target task.

[0113] In addition, historical multimodal data features are used as regularization terms and added to the preset loss function to update the loss value. The gradient value is updated based on the updated loss value. The regularization term is used to constrain the complexity of the target general-purpose model to prevent overfitting. It can also guide the target general-purpose model to maintain a balance between old and new knowledge, thereby improving the generalization ability of the target general-purpose model.

[0114] Furthermore, the regularization term corresponding to the historical multimodal data features can be determined in the form of L2 norm to determine the distance between the target result of the current target general-purpose model and the historical multimodal data features. As part of the loss function, the weight of the regularization term can also be dynamically adjusted according to the time proximity of the historical multimodal data features or their relevance to the current target task, so that the target general-purpose model pays more attention to the recent target task and the key target task.

[0115] In addition, after the preset loss function and gradient value are updated, the preset learning rate is received, and the parameters of the target general-purpose model are updated in combination with optimization algorithms (such as stochastic gradient descent SGD, Adam, RMSprop, etc.). The preset learning rate can be set according to the characteristics of the target task or a dynamic adjustment strategy.

[0116] Specifically, based on the updated loss value, the gradient values ​​of the general-purpose model parameters are determined, the gradients are processed by optimization algorithms (such as momentum acceleration, adaptive learning rate adjustment, etc.), and the general-purpose model parameters are updated in combination with the preset learning rate.

[0117] While updating the parameters of the general model, the historical multimodal data features in the preset memory are updated using the current multimodal data features. The current multimodal data features can be integrated into the preset memory through replacement strategies (such as Least Recently Used LRU) or fusion strategies (such as weighted average) to maintain the timeliness and representativeness of the content in the preset memory.

[0118] Furthermore, the learning rate can be dynamically adjusted using a learning rate scheduler (such as Step Decay, Cosine Annealing, Warmup, etc.) to improve the convergence performance of the target general-purpose model.

[0119] Furthermore, if the current multimodal data consists of multiple sets of multimodal data and the number of multimodal data exceeds a preset threshold, the multimodal data can be divided according to the preset threshold to process the multimodal data in batches, and the general model parameters can be updated based on each batch of multimodal data.

[0120] This embodiment achieves continuous learning and stable performance of the target general-purpose model by introducing historical multimodal data features and dynamically updating the preset memory bank, combined with optimization algorithms and learning rate scheduling strategies. It is suitable for complex application scenarios involving multiple modes and multiple tasks.

[0121] In some embodiments, after monitoring the performance metrics of the trained target general-purpose model according to a preset monitoring period, the method further includes:

[0122] Under the condition that the performance indicators meet the preset performance indicators, monitor the changing trend of the performance indicators over multiple consecutive preset monitoring periods;

[0123] If the performance indicators meet the preset performance indicators within a preset number of consecutive monitoring periods, stop the self-evolution process of the target general-purpose model and save the general-purpose model parameters of the current target general-purpose model.

[0124] If the performance indicators do not meet the preset performance indicators within the preset monitoring period of consecutive preset periods, the trained target general-purpose model will be adjusted according to the preset adjustment method.

[0125] Optionally, in this embodiment, if the performance indicators (such as accuracy, recall, F1 score, inference latency, etc.) of the target general-purpose model are detected to reach or exceed the preset performance indicators within a preset monitoring period, the optimization or self-evolution process of the model will not be stopped immediately. Instead, the changing trend of the performance indicators will be further monitored in the next few consecutive preset monitoring periods. Trend monitoring is used to ensure the stability and continuity of the performance of the target general-purpose model, rather than occasional peak performance.

[0126] Specifically, performance indicators can be recorded within each preset monitoring period to form a performance indicator change curve. By analyzing the performance indicator change curve, it can be determined whether the model performance is continuously stable and gradually improving, or whether there is a fluctuating or declining trend.

[0127] Furthermore, a sliding window mechanism can be used to determine the trend of change. That is, a sliding window of fixed length (e.g., the last 5 monitoring periods) is set, and statistical analysis is performed on the performance indicators within the window, such as calculating the mean, variance, slope, etc., to judge the stability and trend of performance.

[0128] Trend detection algorithms can also be used to determine the changing trend, that is, to use time series analysis methods such as linear regression, exponential smoothing, and moving average to detect the changing trend of performance indicators.

[0129] Furthermore, abnormal fluctuation detection can be used to determine the trend of change, that is, an abnormal detection mechanism is introduced to identify abnormal fluctuations in performance indicators and avoid misjudgment.

[0130] Furthermore, if the performance indicators of the target general-purpose model meet the preset performance indicators for multiple consecutive preset monitoring periods (e.g., 5 consecutive periods), and the performance trend is stable or continuously improving, it can be considered that the target general-purpose model has reached the expected performance standard and has sufficient stability. At this time, the system will automatically stop the model's self-evolution process (such as parameter updates, structural adjustments, etc.) and save the current target general-purpose model parameters.

[0131] At the same time, the performance metrics, training logs, parameter configurations, and other information of the target general-purpose model can be saved for subsequent auditing, analysis, and reproduction.

[0132] Furthermore, model version management can be implemented for the target general-purpose model. This involves using model version control tools (such as MLflow and DVC) to manage the version of the general-purpose model parameters and record each saved model version and its corresponding performance metrics.

[0133] You can also set up an automatic save mechanism, that is, set an automatic save strategy, such as automatically triggering the save operation after the performance indicators have been stable for a certain preset monitoring period, so as to avoid manual intervention.

[0134] Furthermore, when saving general-purpose model parameters, the target general-purpose model can be optimized using model compression techniques (such as quantization and pruning) to reduce storage space and inference latency.

[0135] Furthermore, if the performance indicators of the target communication model fail to meet the preset performance indicators within several consecutive preset monitoring periods, or if the performance shows significant fluctuations or even a downward trend, it is considered that the target communication model has not yet reached a stable state and needs further optimization. In this case, the preset adjustment method for the target communication model will be automatically triggered to adjust the target communication model.

[0136] The preset adjustment methods include structural adjustments to the target general-purpose model and / or adjustments to the update frequency corresponding to the parameters of the general-purpose model. The model structure adjustment can be to increase or decrease the number of layers in the target general-purpose model, adjust the number of heads in the attention mechanism, or modify the network connection method to improve the expressive power or generalization ability of the target general-purpose model.

[0137] Adjusting the update frequency corresponding to the parameters of the general-purpose model can include, but is not limited to, adjusting the learning rate, optimizer type, and batch size to improve the training performance of the target general-purpose model.

[0138] Furthermore, the preset adjustment methods may also include data augmentation strategy adjustment and regularization strategy adjustment. Data augmentation strategy adjustment means introducing new data augmentation methods or adjusting existing augmentation strategies to improve the generalization ability of the target general-purpose model. Regularization strategy adjustment means, for example, adjusting the Dropout ratio or adding new regularization terms to prevent the target general-purpose model from overfitting.

[0139] Furthermore, the target general-purpose model can be adjusted in the following ways:

[0140] Automatic hyperparameter optimization involves automatically searching for the optimal combination of hyperparameters using methods such as Bayesian optimization, genetic algorithms, and reinforcement learning. Neural architecture search (NAS) involves automatically searching for the optimal model structure to improve model performance. Online learning strategy adjustment involves dynamically adjusting parameters such as learning rate and batch size based on performance feedback to achieve adaptive optimization of the target general-purpose model.

[0141] This embodiment achieves the ability to continuously monitor the performance indicators of the target general-purpose model, automatically determine the timing for terminating the optimization of the target general-purpose model, and automatically adjust the target general-purpose model when necessary, thus ensuring the stability and reliability of the performance indicators of the target general-purpose model.

[0142] To effectively address the shortcomings of traditional technologies in terms of insufficient synergy between general and specialized knowledge, and to significantly improve the synergistic effect and update efficiency of multimodal general-specific models based on new knowledge, this application provides an embodiment of an incremental learning-based multimodal general-specific model self-evolution system for implementing all or part of the aforementioned incremental learning-based multimodal general-specific model self-evolution. See [link to embodiment]. Figure 2 The aforementioned self-evolutionary system for a multimodal communication model based on incremental learning specifically includes the following components:

[0143] The first processing module 10 is used to receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general-purpose model corresponding to the target task based on the target task. The target general-purpose model includes a target general model and a target specific model.

[0144] The second processing module 20 is used to extract data features of multimodal data through the target general model to obtain multimodal data features, and to process the multimodal data features and multimodal data through the target specific model to obtain the target result corresponding to the target task;

[0145] The update module 30 is used to obtain the historical multimodal data features corresponding to the historical multimodal data, and update the parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data and the target result.

[0146] The adjustment module 40 is used to monitor the performance indicators of the trained target general-purpose model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, the trained target general-purpose model is adjusted according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

[0147] As described above, the multimodal general-purpose model self-evolution system based on incremental learning provided in this application can innovatively receive the current target task and the multimodal data corresponding to the current target task, determine the trained target general-purpose model corresponding to the target task based on the target task, extract data features of multimodal data through the target general model in the target general-purpose model to obtain multimodal data features, generate target results by combining the multimodal data features and multimodal data through the target specific model in the target general-purpose model, and update the general-purpose model of the target general-purpose model based on historical multimodal data features, current multimodal data, and target results. The method monitors the performance metrics of the trained target general-specific model according to a preset monitoring period. If the performance metrics corresponding to the current preset monitoring period do not meet the preset performance metrics, the trained target general-specific model is adjusted according to a preset adjustment method. This allows for dynamic adjustment of the target general-specific model's structure and parameters through incremental learning and collaborative updates, reducing the need for full retraining. It also improves the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-specific model's ability to extract features from multimodal data and adapt to target tasks, improving the accuracy and robustness of the target results. This method effectively addresses the shortcomings of traditional techniques in terms of insufficient synergy between general and specialized knowledge, significantly improving the synergistic effect of multimodal general-specific models and the update effect based on new knowledge.

[0148] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in terms of insufficient synergy between general and specialized knowledge, and to significantly improve the synergistic effect of multimodal general-specific models and the updating effect based on new knowledge, this application provides an embodiment of an electronic device for implementing all or part of the aforementioned incremental learning-based self-evolution method for multimodal general-specific models. The electronic device specifically includes the following components:

[0149] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between a multimodal communications-specific model self-evolution system based on incremental learning and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to an embodiment of a multimodal communications-specific model self-evolution method based on incremental learning and an embodiment of a multimodal communications-specific model self-evolution system based on incremental learning, the contents of which are incorporated herein by reference, and repeated details will not be described again.

[0150] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0151] In practical applications, a portion of a self-evolution method for a multimodal communication model based on incremental learning can be executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0152] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0153] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0154] In one embodiment, a self-evolution method for a multimodal general-purpose model based on incremental learning can be integrated into a central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0155] Step S101: Receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general-purpose model corresponding to the target task based on the target task. The target general-purpose model includes a target general model and a target specific model.

[0156] Step S102: Extract data features of multimodal data through the target general model to obtain multimodal data features, process the multimodal data features and multimodal data through the target specific model to obtain the target result corresponding to the target task;

[0157] Step S103: Obtain the historical multimodal data features corresponding to the historical multimodal data, and update the general-specific model parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result;

[0158] Step S104: Monitor the performance indicators of the trained target general-purpose model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target general-purpose model according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

[0159] As described above, the electronic device provided in this application innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target general-purpose model corresponding to the target task based on the target task, extracts data features of the multimodal data through the target general model in the target general-purpose model, obtains multimodal data features, generates target results by combining the multimodal data features and multimodal data through the target specific model in the target general-purpose model, updates the general-purpose model parameters of the target general-purpose model based on historical multimodal data features, current multimodal data, and target results, monitors the performance indicators of the trained target general-purpose model according to a preset monitoring cycle, and adjusts the trained target general-purpose model according to a preset adjustment method if the performance indicators corresponding to the current preset monitoring cycle do not meet the preset performance indicators. It can dynamically adjust the structure and parameters of the target general-purpose model through incremental learning and collaborative updating of the target general-purpose model, reducing the need for full retraining, improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-purpose model's feature extraction and adaptability to target tasks, and improving the accuracy and robustness of the target results. This method effectively addresses the shortcomings of traditional technologies in terms of insufficient synergy between general and specialized knowledge, and significantly improves the synergistic effect of multimodal general and specialized models and the updating effect based on new knowledge.

[0160] In another embodiment, a multimodal general-purpose model self-evolution system based on incremental learning can be configured separately from the central processing unit 9100. For example, a multimodal general-purpose model self-evolution system based on incremental learning can be configured as a chip connected to the central processing unit 9100, and the function of a multimodal general-purpose model self-evolution method based on incremental learning can be realized through the control of the central processing unit.

[0161] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0162] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0163] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0164] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0165] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0166] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0167] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0168] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0169] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the self-evolution method for a multimodal communication-specific model based on incremental learning, where the execution subject is a server or client, as described in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the self-evolution method for a multimodal communication-specific model based on incremental learning, where the execution subject is a server or client, as described in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0170] Step S101: Receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general-purpose model corresponding to the target task based on the target task. The target general-purpose model includes a target general model and a target specific model.

[0171] Step S102: Extract data features of multimodal data through the target general model to obtain multimodal data features, process the multimodal data features and multimodal data through the target specific model to obtain the target result corresponding to the target task;

[0172] Step S103: Obtain the historical multimodal data features corresponding to the historical multimodal data, and update the general-specific model parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result;

[0173] Step S104: Monitor the performance indicators of the trained target general-purpose model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target general-purpose model according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

[0174] As described above, the computer-readable storage medium provided in this application innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target general-purpose model corresponding to the target task based on the target task, extracts data features of the multimodal data through the target general model in the target general-purpose model, obtains multimodal data features, generates target results by combining the multimodal data features and multimodal data through the target specific model in the target general-purpose model, updates the general-purpose model parameters of the target general-purpose model based on historical multimodal data features, current multimodal data, and target results, monitors the performance indicators of the trained target general-purpose model according to a preset monitoring period, and adjusts the trained target general-purpose model according to a preset adjustment method if the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators. It can dynamically adjust the structure and parameters of the target general-purpose model through incremental learning and collaborative updating of the target general-purpose model, reducing the need for full retraining, improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-purpose model's feature extraction and adaptability to target tasks, and improving the accuracy and robustness of the target results. This method effectively addresses the shortcomings of traditional technologies in terms of insufficient synergy between general and specialized knowledge, and significantly improves the synergistic effect of multimodal general and specialized models and the updating effect based on new knowledge.

[0175] Embodiments of this application also provide a computer program product capable of implementing all steps of the self-evolution method for a multimodal general-purpose model based on incremental learning, where the execution subject is a server or client as described in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the self-evolution method for a multimodal general-purpose model based on incremental learning. For example, the computer program / instruction implements the following steps:

[0176] Step S101: Receive the current target task and the multimodal data corresponding to the current target task, and determine the trained target general-purpose model corresponding to the target task based on the target task. The target general-purpose model includes a target general model and a target specific model.

[0177] Step S102: Extract data features of multimodal data through the target general model to obtain multimodal data features, process the multimodal data features and multimodal data through the target specific model to obtain the target result corresponding to the target task;

[0178] Step S103: Obtain the historical multimodal data features corresponding to the historical multimodal data, and update the general-specific model parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result;

[0179] Step S104: Monitor the performance indicators of the trained target general-purpose model according to the preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, adjust the trained target general-purpose model according to the preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

[0180] As described above, the computer program product provided in this application innovatively receives the current target task and the multimodal data corresponding to the current target task, determines the trained target general-purpose model corresponding to the target task based on the target task, extracts data features of the multimodal data through the target general model in the target general-purpose model, obtains multimodal data features, generates target results by combining the multimodal data features and multimodal data through the target specific model in the target general-purpose model, updates the general-purpose model parameters of the target general-purpose model based on historical multimodal data features, current multimodal data, and target results, monitors the performance indicators of the trained target general-purpose model according to a preset monitoring cycle, and adjusts the trained target general-purpose model according to a preset adjustment method if the performance indicators corresponding to the current preset monitoring cycle do not meet the preset performance indicators. It can dynamically adjust the structure and parameters of the target general-purpose model through incremental learning and collaborative updating of the target general-purpose model, reducing the need for full retraining, while improving the ability to balance new and old knowledge, avoiding catastrophic forgetting, thereby enhancing the target general-purpose model's feature extraction and adaptability to target tasks, and improving the accuracy and robustness of the target results. This method effectively addresses the shortcomings of traditional technologies in terms of insufficient synergy between general and specialized knowledge, and significantly improves the synergistic effect of multimodal general and specialized models and the updating effect based on new knowledge.

[0181] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0185] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A self-evolution method for multimodal general knowledge models based on incremental learning, characterized in that, The method includes: Receive the current target task and the multimodal data corresponding to the target task, determine and execute the trained target general-purpose model corresponding to the target task based on the target task, the target general-purpose model includes a target general model and a target specific model, wherein the multimodal data includes at least two of image data, text data and audio data; The multimodal data features are extracted using the target general model to obtain multimodal data features. The multimodal data features and the multimodal data are then processed using the target specific model to obtain the target result corresponding to the target task. Obtain the historical multimodal data features corresponding to the historical multimodal data, and update the parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result; The performance indicators of the trained target general-purpose model are monitored according to a preset monitoring cycle. If the performance indicators corresponding to the current preset monitoring cycle do not meet the preset performance indicators, the trained target general-purpose model is adjusted according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

2. The method according to claim 1, characterized in that, After receiving the current target task and the multimodal data corresponding to the current target task, the method further includes: Data cleaning is performed on the multimodal data corresponding to the current target task to obtain cleaned multimodal data; The cleaned multimodal data is mapped to a preset scale range to obtain normalized multimodal data.

3. The method according to claim 1, characterized in that, The target-specific model also includes a fusion mechanism; the process of processing multimodal data features and the multimodal data through the target-specific model to obtain the target result corresponding to the target task includes: The multimodal data features and the multimodal data are processed by the target-specific model to obtain specific features; Based on the fusion mechanism, a general correlation index between the multimodal data features and the current target task, and a specific correlation index between the specific features and the current target task are determined. Based on the general correlation index and the specific correlation index, a general weight corresponding to the multimodal data features and a specific weight corresponding to the specific features are determined. Based on the general weights and the specific weights, the multimodal data features and the specific features are fused to obtain the target result corresponding to the target task.

4. The method according to claim 1, characterized in that, The process of updating the parameters of the trained target-specific model based on historical multimodal data features, current multimodal data, and target results includes: For the current multimodal data, receive the feedback result of the current target result, correct the target result based on the feedback result, and obtain the corrected target result; The loss value between the corrected target result and the target result is determined by a preset loss function, and the gradient value of the current target general-purpose model is determined based on the loss value, so as to determine the update direction of the general-purpose model parameters of the target general-purpose model; Receive a preset learning rate, and update the parameters of the general-purpose model based on the optimization algorithm, the gradient value, the historical multimodal data features, and the preset learning rate.

5. The method according to claim 4, characterized in that, The receiving of a preset learning rate, and updating the parameters of the general-purpose model based on the optimization algorithm, the gradient value, the historical multimodal data features, and the preset learning rate, includes: Extract the historical multimodal data features from the preset memory, use the historical multimodal data features as the regularization term of the preset loss function, and update the loss value and the gradient value; The system receives a preset learning rate, updates the parameters of the general-purpose model based on the optimization algorithm, the updated gradient values, and the preset learning rate, and updates the historical multimodal data features in the preset memory based on the current multimodal data features.

6. The method according to claim 1, characterized in that, After monitoring the performance metrics of the trained target communication model according to the preset monitoring period, the method further includes: When the performance index meets the preset performance index, monitor the trend of the performance index over multiple consecutive preset monitoring periods; If the performance indicators meet the preset performance indicators within the preset monitoring period for a consecutive preset number of periods, the self-evolution process of the target communication model is stopped, and the communication model parameters of the current target communication model are saved. If the performance indicators do not meet the preset performance indicators within the preset monitoring period for a consecutive preset number of periods, the trained target general-purpose model shall be adjusted according to the preset adjustment method.

7. A self-evolving system for a multimodal general knowledge model based on incremental learning, characterized in that, The system includes: The first processing module is configured to receive the current target task and the multimodal data corresponding to the current target task, and determine the target general-purpose model corresponding to the target task to be executed based on the target task. The target general-purpose model includes a target general model and a target specific model. The multimodal data includes at least two of image data, text data and audio data. The second processing module is used to extract data features of multimodal data through the target general model to obtain multimodal data features, and to process the multimodal data features and the multimodal data through the target specific model to obtain the target result corresponding to the target task; The update module is used to obtain the historical multimodal data features corresponding to the historical multimodal data, and update the parameters of the trained target general-specific model based on the historical multimodal data features, the current multimodal data, and the target result. The adjustment module is used to monitor the performance indicators of the trained target general-purpose model according to a preset monitoring period. If the performance indicators corresponding to the current preset monitoring period do not meet the preset performance indicators, the trained target general-purpose model is adjusted according to a preset adjustment method. The preset adjustment method includes adjusting the structure of the trained target general-purpose model and / or adjusting the update frequency corresponding to the parameters of the general-purpose model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the self-evolution method for a multimodal communication model based on incremental learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the self-evolution method for a multimodal communication model based on incremental learning as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the self-evolution method for a multimodal communication model based on incremental learning as described in any one of claims 1 to 6.

Images