Data processing method and device based on adaptive meta-learning, equipment and medium

Abstract

The present application relates to the technical field of artificial intelligence, which can be applied to business scenarios such as financial technology and medical health, and discloses a data processing method and device based on adaptive meta-learning, equipment and medium, comprising: acquiring multi-modal input data and extracting initial modal features, generating modal confidence scores using a gating network, and weighting the initial modal features to obtain weighted modal features; generating intra-modal prototypes according to the weighted modal features and weighting and fusing them into cross-modal prototypes; encoding the current task, retrieving meta-parameters as initial values of the model, constructing a task-specific classifier and a meta-classifier, updating the parameters of the task-specific classifier and the meta-classifier based on the cross-modal prototypes, processing the target task using the updated task-specific classifier, and outputting the result. The present application improves the information utilization efficiency between modalities through confidence-driven modal weighting and prototype fusion guided by fusion weight, and improves the speed and accuracy of model adaptation to new tasks by combining task encoding with optimal initialization parameters.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a data processing method and device based on adaptive meta-learning, equipment and a storage medium. BACKGROUND

[0002] In the multi-modal small sample learning task, the existing technology generally has the problems of insufficient modeling of modal heterogeneity, weak task adaptation ability, and limited generalization performance, especially in the financial technology business field and the medical health business field.

[0003] In the medical health business field, in the face of multi-source heterogeneous data such as medical images, electronic medical record texts, and voice inquiry records, the existing multi-modal small sample model generally uses a fixed architecture feature extractor and performs simple splicing or average fusion processing on each modal data, which cannot dynamically adjust the influence of different modalities in model decision-making according to the specific needs of the task. This static processing method is easy to cause the amplification of part of the weak modal information or the dilution of the strong modal information in the medical scene, thereby causing the risk of misdiagnosis, especially in the actual clinical application with extremely small sample quantity.

[0004] In the financial technology business field, the correlation between modalities such as financial images, transaction texts, and voice customer service data varies with the task. The existing model usually uses a preset structure for feature fusion, lacking dynamic perception ability for the importance of modalities, resulting in problems such as modal redundancy enhancement or key modal suppression in tasks such as anomaly detection and identity recognition, thereby affecting the accuracy of model judgment. In addition, when the financial task frequently switches (such as from fraud identification task to risk assessment task), the existing method is difficult to quickly select or adjust the parameter initialization strategy according to the new task, affecting the rapid adaptation ability under the condition of small sample.

[0005] Traditional meta-learning methods have been tried to be combined with multi-modal pre-training models, but most technical solutions have failed to effectively build a task adaptation mechanism. When dealing with cross-modal task switching, such as from visual question answering to multi-modal text classification, the existing model cannot fully mobilize the meta-knowledge in the historical task, resulting in slow response to new tasks and more dependence on labeled samples in the learning process. In addition, under the condition of small sample, the existing model often uses a fixed initialization strategy for parameter fine-tuning, lacking a dynamic parameter retrieval mechanism driven by task encoding, and is prone to problems such as large initialization bias and low convergence efficiency. SUMMARY

[0006] The main purpose of the present application is to provide a data processing method, device and equipment based on adaptive meta-learning and a storage medium, aiming to solve the technical problem that the prior art lacks a mechanism for dynamically determining model initialization parameters and inter-modal dynamic weight distribution based on task characteristics in multi-modal small sample learning, resulting in poor adaptability and weak generalization ability in task switching and modal heterogeneity scenarios.

[0007] To achieve the above-mentioned purpose, the present application provides a data processing method based on adaptive meta-learning, comprising:

[0008] Obtain input data of different modalities, and extract initial modal features of the input data of each modality respectively;

[0009] Based on the initial modal features, generate a modal confidence score through a gating network, and perform weighted processing on the initial modal features based on the modal confidence score to obtain weighted modal features;

[0010] According to the weighted modal features, generate intra-modal prototypes corresponding to each modality, and weight and fuse multiple intra-modal prototypes based on a fusion weight coefficient to generate cross-modal prototypes;

[0011] Encode the current learning task to obtain a task embedding vector, and in a preset meta-parameter pool, retrieve a target meta-parameter according to the task embedding vector, and use the target meta-parameter as an initial value of a model parameter;

[0012] Establish a task-specific classifier and a meta-classifier using the initial value of the model parameter, and update the parameters of the task-specific classifier and the meta-classifier based on the cross-modal prototypes;

[0013] Process a target task through the updated task-specific classifier to generate a target task processing result.

[0014] Further, to achieve the above-mentioned purpose, the present application provides a data processing device based on adaptive meta-learning, comprising:

[0015] A multi-modal feature extraction module is configured to obtain input data of different modalities, and extract initial modal features of the input data of each modality respectively;

[0016] A modal adaptive weighting module is configured to generate a modal confidence score through a gating network based on the initial modal features, and perform weighted processing on the initial modal features based on the modal confidence score to obtain weighted modal features;

[0017] A cross-modal prototype fusion module is configured to generate intra-modal prototypes corresponding to each modality according to the weighted modal features, and weight and fuse multiple intra-modal prototypes based on a fusion weight coefficient to generate cross-modal prototypes;

[0018] a task-adaptive parameter retrieval module configured to encode a current learning task to obtain a task embedding vector, and retrieve a target meta-parameter from a preset meta-parameter pool according to the task embedding vector, and use the target meta-parameter as an initial value of a model parameter;

[0019] a classifier joint training module configured to establish a task-specific classifier and a meta-classifier using the initial value of the model parameter, and update parameters of the task-specific classifier and the meta-classifier based on the cross-modal prototype;

[0020] a target task reasoning module configured to process a target task by using the updated task-specific classifier to generate a target task processing result.

[0021] Further, to achieve the above object, the present application also provides a computer device, which comprises a memory, a processor and a data processing program based on adaptive meta-learning stored in the memory and executable on the processor, and the data processing program based on adaptive meta-learning, when executed by the processor, implements the steps of the data processing method based on adaptive meta-learning.

[0022] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a data processing program based on adaptive meta-learning, and the data processing program based on adaptive meta-learning, when executed by a processor, implements the steps of the data processing method based on adaptive meta-learning.

[0023] Beneficial effects: The application relates to the technical field of artificial intelligence, can be applied to business scenarios such as financial technology and medical health, and discloses a data processing method, device and equipment based on adaptive meta-learning and a medium, which comprises the following steps: acquiring multi-modal input data and extracting initial modal features, generating modal confidence scores by using a gating network and performing weighted processing on the initial modal features to obtain weighted modal features; generating an intra-modal prototype according to the weighted modal features, performing weighted fusion on the intra-modal prototype by using a fusion weight coefficient to obtain a cross-modal prototype; encoding a current learning task to obtain a task embedding vector, and retrieving a target meta-parameter from a meta-parameter pool to initialize model parameters; constructing a task-specific classifier and a meta-classifier based on the initialized parameters, performing parameter updating based on the cross-modal prototype, processing a target task by using the updated task-specific classifier, and outputting a final processing result. The application introduces a modal confidence score in the feature extraction stage, realizes dynamic weighting of multi-modal input data, effectively reduces modal noise interference, introduces a fusion weight coefficient in the prototype generation and fusion stage, enhances the adaptation ability between modes, encodes a learning task and retrieves a meta-parameter of the most similar task as initialization, significantly improves the rapid adaptation ability and convergence speed of the model under a new task, and therefore realizes higher task generalization performance and robustness in a multi-modal small sample scene. BRIEF DESCRIPTION OF DRAWINGS

[0024] The application will be further described below in combination with the drawings and embodiments, and the drawings are as follows:

[0025] Figure 1 An application environment schematic diagram of the data processing method based on adaptive meta-learning in an embodiment of the application;

[0026] Figure 2 A flow schematic diagram of the data processing method based on adaptive meta-learning in an embodiment of the application;

[0027] Figure 3 A function module schematic diagram of the data processing device based on adaptive meta-learning in a preferred embodiment of the application;

[0028] Figure 4 A structure schematic diagram of a computer device in an embodiment of the application;

[0029] Figure 5 Another structure schematic diagram of a computer device in an embodiment of the application. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are merely intended to explain the application, and are not intended to limit the application.

[0031] The data processing method based on adaptive meta-learning provided in the embodiments of the application can be applied to business scenarios such as Figure 1The application environment of the application is that the user end communicates with the service end through the network. The service end can obtain multi-modal input data through the user end and extract initial modal features, generate modal confidence scores by using a gating network, and perform weighted processing on the initial modal features to obtain weighted modal features; generate intra-modal prototypes according to the weighted modal features, perform weighted fusion on the intra-modal prototypes by using a fusion weight coefficient to obtain cross-modal prototypes; encode a current learning task to obtain a task embedding vector, and retrieve a target meta-parameter from a meta-parameter pool to initialize model parameters; construct a task-specific classifier and a meta-classifier based on the initialized parameters and update the parameters based on the cross-modal prototypes, process a target task by using the updated task-specific classifier, and output a final processing result. By introducing the modal confidence score in the feature extraction stage, the application realizes dynamic weighting of the multi-modal input data, effectively reduces modal noise interference; by introducing the fusion weight coefficient in the prototype generation and fusion stage, the adaptation ability between modalities is enhanced; by encoding the learning task and retrieving the meta-parameter of the most similar task as the initialization, the rapid adaptation ability and convergence speed of the model under the new task are significantly improved, thereby realizing higher task generalization performance and robustness in the multi-modal small sample scene. The user end can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The service end can be realized by an independent server or a server cluster composed of multiple servers. The application will be described in detail below through specific embodiments.

[0032] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the data processing method based on adaptive meta-learning provided by the application is shown. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0033] As Figure 2 shown, the data processing method based on adaptive meta-learning provided by the application includes the following steps:

[0034] S10, obtaining input data of different modalities and extracting initial modal features of the input data of each modality;

[0035] In this embodiment, in order to realize efficient fusion of multi-modal information and task adaptability enhancement, it is necessary to accurately obtain the input of various heterogeneous data sources in the model processing process and unify it into a structured modal feature vector. The input data of different modalities includes image data, text data, audio data, etc. These data have significant differences in content expression, structure form and signal dimension, and direct processing will lead to ineffective information fusion or weakened representation ability. Therefore, it is necessary to first process the input of each modality by channel, extract its intrinsic features, and complete dimension unification.

[0036] Image data as typical two-dimensional structure data, its processing can be carried out by constructing a convolutional neural network extractor to extract multi-scale spatial features. The convolution kernel extracts local region texture, edge and shape information in the sliding window. Through multiple convolution and pooling layer stacking, the feature tensor of image mode is output. Common models such as ResNet or EfficientNet have high throughput feature expression ability. Text data has time sequence and semantic nesting, and a pre-trained language model can be used as a feature extraction unit to embed the segmented sentence or paragraph into a high-dimensional semantic space to obtain a context representation for downstream processing. Specific implementation can be carried out by BERT or RoBERTa, and the output is a fixed-dimensional semantic vector. Audio data is a continuous time domain signal, which can be converted into a two-dimensional time-frequency matrix similar to an image by short-time Fourier transform to extract its frequency spectrum representation, and then input into a spectral analysis network, such as using a voiceprint feature extractor based on 1D or 2D convolution structure, to output a high-dimensional vector of audio mode.

[0037] After the above image, text and audio modal feature extraction is completed, due to the obvious difference in output dimension and distribution of each mode, in order to ensure the numerical stability and representation consistency in the subsequent fusion step, normalization operation needs to be performed on different modal features. The normalization process can use LayerNorm or BatchNorm mechanism to make each feature dimension have similar mean and variance performance in numerical scale, so as to suppress the gradient imbalance problem between modes. Further, the processed modal features are projected into the same dimensional space, which is completed by a linear transformation matrix. The image features, text features and audio features are multiplied by their respective conversion matrices respectively, and a learnable bias term is added, so as to unify the modal vector set with compatible structure, which is convenient for subsequent fusion and dynamic weighting process.

[0038] The channel network parameters in the feature extraction process can be initialized according to the pre-trained model, or introduced into the task-related field model through transfer learning. In order to enhance the robustness of the processing, a Dropout layer can be added at the output end of the channel to reduce the risk of overfitting, or a residual connection can be introduced to improve the efficiency of information flow.

[0039] The image feature extraction process can adopt a 50-layer ResNet structure as the visual coding network, and the input image is uniformly adjusted to a resolution of 224x224. Different levels of visual semantic information are extracted through five residual blocks, and the output dimension is set to 1024 dimensions. The text processing channel can be based on the RoBERTa-large model, and the input text is Tokenize processed. The maximum length is set to 128, and the [CLS] vector is obtained through the Transformer structure as the text modal representation. The final output vector dimension is 1024 dimensions. The audio input uses original speech data with a sampling rate of 16 kHz. A window function with a sliding window length of 25 ms and a step length of 10 ms is used to generate a Mel spectrum graph. The input is input to a 5-layer 2D convolution structure, and the final output dimension is also set to 1024 dimensions.

[0040] The normalization process uniformly uses the LayerNorm operation to process the output vector of each modality; the dimension uniformly uses a shared linear mapping layer to input the feature tensor of different modalities and map to a 512-dimensional unified space. In the training stage, the linear mapping matrix is used as a learnable parameter to participate in gradient update.

[0041] The number of modal channels can also be adjusted according to the actual application scenario requirements, for example, in scenarios where speech information is not required, the audio channel processing can be omitted, and only the image and text two-way input is retained. Time series data such as sensor output can also be added as a new modality to build an independent channel for processing and then unified to the feature space. The preprocessing method of each modality input can be adjusted according to the actual distribution characteristics of the data source, for example, the CT image in medical imaging can use CLAHE contrast enhancement as preprocessing, and the financial text in financial report data can introduce entity recognition mechanism to enhance semantic density.

[0042] Example: In the medical health business field, patient medical record text, medical image and physical sign monitoring speech data can be used as multi-modal input, and channel processing and unified vector mapping are performed on them to realize comprehensive modeling of individual health status and quickly complete disease screening classification under the condition of few sample labels.

[0043] In the financial technology business field, company financial report text, stock trend image and expert voice comment can be used as multi-modal data source input into the system to complete investment risk level prediction task, solve the analysis deviation caused by the semantic misalignment between modalities, and improve the cross-modal judgment accuracy and flexibility.

[0044] The embodiment can effectively solve the problem of large difference in expression structure between heterogeneous modalities by independent channel processing and dimension unified modeling of different types of modal input data, while maintaining the integrity of the information of each modality, realizing semantic alignment and scale consistency of subsequent fusion operation. After introducing the normalization and linear transformation mechanism, the model can maintain stable feature extraction effect under the condition of task switching or insufficient sample size, and reduce the precision loss caused by the representation shift between modalities.

[0045] S20, based on the initial modality feature, generating a modality confidence score through a gating network, and weighting the initial modality feature based on the modality confidence score to obtain a weighted modality feature;

[0046] In the embodiment, after completing the initial modality feature extraction and dimension unified processing of the multi-modal input data, in order to further enhance the dynamic adaptation ability of each modality under different tasks or samples, a modality confidence modeling mechanism needs to be introduced to dynamically weight the importance of each modality feature. This process relies on further analysis of the initial modality feature, and uses a gating network to give each type of modality feature an adaptive importance score to measure the influence of the modality on the decision result in the current context, and then guide the weighting processing operation.

[0047] The gating network takes the initial modality feature as input and constructs a learnable shallow subnetwork, which usually includes linear transformation, bias adjustment, activation function processing and normalization operation. In the linear transformation process, the feature vector of each modality is first multiplied by the corresponding weight matrix to change or reorganize the structure of the feature space; then a learnable bias term is added to enhance the expression freedom of the model. A nonlinear activation function such as ReLU, LeakyReLU or GELU is applied to the transformed result to introduce nonlinear feature selection capability to avoid the expression bottleneck of linear transformation. The activation output then enters the normalization unit, which can use Softmax or Sigmoid to normalize it, so that the confidence scores of all modalities fall within the same scale and have the characteristics of probability or weight sum of 1.

[0048] The generated modality confidence score is then multiplied element by element with the corresponding initial modality feature to complete the weighting processing. This weighting operation has the ability to retain useful information and suppress redundant features, and can automatically attenuate the influence of low-quality modalities or noisy modalities and strengthen the contribution of high-information-density modalities. In the implementation process, if the initial feature dimension of each modality is D, the confidence score output by the gating network can be a scalar (representing the overall weight) or a D-dimensional vector (used to weight each dimension of the feature), depending on the task requirements and computing resources.

[0049] In the case of multiple modalities coexisting, the feature vectors of each modality are respectively subjected to confidence calculation and weighting processing to form a weighted feature vector of each modality. Finally, the weighted features of all modalities are spliced, stacked or kept independent channels for subsequent fusion operation.

[0050] For image modality features, a three-layer fully connected gating network structure can be used, where each layer includes linear transformation, BatchNorm and ReLU activation, and the last layer uses Sigmoid to output a single-channel weight coefficient. The text modality and the audio modality can be processed by using the gating network with the same structure, but the weight matrix and the bias term are independently trained to realize the reservation of the expression difference between modalities.

[0051] The training of the gating network can be jointly performed with the overall model, and the output of the module does not need to be separately constrained in the loss function. Only indirect supervised learning is performed through the total task loss. The KL divergence or entropy regular term can also be introduced to control the discreteness of the confidence of each modality, so as to avoid the model from being excessively dependent on a certain modality.

[0052] In actual deployment, to reduce the amount of calculation, the gating network can be simplified to a single-layer linear layer and directly output a scalar confidence. For an environment with strong computing power, a multi-head structure can be used for the confidence output, and independent weight coefficients are generated for each feature dimension to enhance the dynamic control ability of local information.

[0053] In the deployment of a heterogeneous system, the gating network can be encapsulated as an independent service unit, only inputting a modality feature tensor and outputting a confidence vector, and realizing distributed weighting processing through a remote calling interface to adapt to cloud-edge collaboration or terminal reasoning tasks.

[0054] By introducing the gating network based on feature driving, the embodiment adaptively generates modality confidence scores and performs feature-level weighting processing, which can dynamically adjust the weight distribution of different modalities in different input tasks, thereby effectively improving the robustness of the model to modality heterogeneity and uncertainty.

[0055] S30, generating a modal intra prototype corresponding to each modality according to the weighted modality feature, and generating a cross-modal prototype by weightedly fusing the multiple modal intra prototypes based on the fusion weight coefficient;

[0056] In the embodiment, after completing the weighting processing of the multi-modal input data, it is still necessary to further establish the expression uniformity between modalities to solve the problem of inconsistent semantic spaces of different modalities. Therefore, the modal intra prototype is constructed based on the weighted modality feature, and the cross-modal prototype is generated through a fusion mechanism for subsequent task modeling. The prototype construction emphasizes the statistical expression of the intra-modality samples, and the fusion operation focuses on the controllable combination of the contributions between modalities.

[0057] The generation of intra-modal prototype refers to the aggregation of all weighted modal features in the same modal, the same category or the same task context. The arithmetic mean is often used to achieve this. For a certain modal, let its weighted feature be represented as The prototype vector p m of this modal can be defined as:

[0058]

[0059] This average operation can effectively compress the feature distribution of the same category, eliminate noise disturbance and extract the stable center expression within the modal. The calculation of intra-modal prototype can be performed within a specific task sample or updated in the cross-task accumulated sample representation to enhance robustness.

[0060] After obtaining the intra-modal prototypes of each modal, a fusion weight coefficient is further introduced to perform weighted fusion operation on different modal prototypes to construct cross-modal prototype expression. This fusion operation not only integrates the global information of intra-modal prototypes, but also dynamically adjusts the fusion proportion of different modal prototypes. The fusion weight coefficient can be statically preset or dynamically generated by the upstream module or meta-learning mechanism. In actual calculation, it can be represented as:

[0061]

[0062] Where α m is the fusion weight coefficient of the mth modal, satisfying ∑ m α m = 1. If a dynamic weight mechanism is used, the weight can be generated by analyzing the task embedding or modal context through another gating module.

[0063] The weighting in the fusion process can use element-wise multiplication, or first concatenate all modal prototypes and then introduce a weight matrix for linear combination. In order to avoid the conflict of common information between modalities, a modal orthogonality constraint or normalization mechanism can be introduced to ensure the stability and discriminability of the cross-modal prototype.

[0064] In the implementation process, intra-modal prototype calculation and weighted fusion can be implemented as independent network modules, supporting batch processing and gradient backpropagation; at the same time, they can also be packaged as a prototype generation layer, providing configurable modal fusion strategies and weight configuration interfaces.

[0065] In a class of image-text-audio multi-modal tasks, independent intra-modal prototype calculation modules can be trained for each type of modal, and a unified fusion weight generation network is shared. This network receives task encoding or modal confidence information and outputs the corresponding fusion weight coefficient vector. For low-power scenarios, fixed weights can also be preset, such as 0.4 for image modal, 0.4 for text modal and 0.2 for audio modal.

[0066] In a system with a high application level, the fusion weight generation can be a learnable subnetwork related to the task embedding, driven by the task embedding vector, and outputting different modal fusion proportions. For example, when the system identifies that the task belongs to the image-text matching type, the fusion weight will adaptively increase the image and text modal weight and reduce the influence of the audio modal.

[0067] For the case of inconsistent modal feature dimensions, linear mapping can be performed on each modal feature to a unified dimensional space before the intra-modal prototype is generated, and the intra-modal prototype is uniformly processed during the prototype fusion process. An attention mechanism can also be introduced to weight the intra-modal prototype before fusion, enhancing the dynamic interaction capability between modalities.

[0068] Example: In a medical health business scenario, for a diagnosis task containing X-ray images, medical record texts, and doctor's voice, the intra-modal prototype can express image lesion morphology features, text symptom description features, and tension emotion features in the voice tone, respectively. Through weighted fusion by the fusion weight, the stability and accuracy of the overall model can be improved when a certain modal is missing or abnormal.

[0069] In a financial technology business scenario, when processing a task containing stock market trend images, news voice broadcast, and policy text analysis, the fusion proportion of each modal prototype can be dynamically adjusted according to different task scenarios. For example, in the event handling of market fluctuations, the importance of the image modal prototype is increased, while in the policy analysis task, the text modal prototype may dominate, and the fusion mechanism supports smooth transition when the task changes.

[0070] By constructing intra-modal prototypes for weighted modal features and introducing fusion weight coefficients for weighted fusion, the embodiment can realize structural compression and global integration of modal information, enhance the semantic consistency and expression stability of multi-modal features in downstream tasks. The intra-modal prototype retains the class center information specific to the modal, while the cross-modal prototype improves the fault tolerance to modal redundancy and heterogeneity through fusion strategies, thereby improving the discriminability and generalization performance of the model under small sample and unbalanced data.

[0071] S40, encode the current learning task to obtain a task embedding vector, and in a preset meta-parameter pool, retrieve a target meta-parameter according to the task embedding vector, and take the target meta-parameter as an initial value of the model parameter;

[0072] In the embodiment, in order to realize efficient migration and rapid adaptation between multi-modal tasks, the feature information of the current learning task needs to be encoded to obtain a task embedding vector that can represent the characteristics of the task, and a retrieval operation is performed in a preset meta-parameter pool based on the task embedding vector, so as to obtain the model parameters corresponding to the historical task most similar to the current task as the initialization value of the current model, thereby improving the training efficiency and enhancing the cross-task generalization capability.

[0073] The task encoding operation usually takes task-related context information as input, such as task name, task input modality type, sample feature distribution, class number, inter-class distance, sample mean and variance, distribution characteristics of class prototypes, etc. By designing a task encoder module, such as a multi-layer perceptron, an attention network or a graph neural network structure, these task feature inputs are mapped to a vector representation in a low-dimensional continuous space, i.e., a task embedding vector. The task embedding vector has the ability to describe the complexity of the task structure, the modality characteristics and the sample distribution pattern.

[0074] After the task embedding vector is constructed, a meta-parameter pool is introduced as a knowledge repository to support task initialization. The meta-parameter pool stores the task embedding vectors and corresponding model parameters of multiple historical tasks. Each set of parameters is extracted after being optimized and trained on a certain historical task, and has pertinence and stability.

[0075] The core of the retrieval process is to determine the most matched meta-task based on the similarity measurement between the task embedding vector and all historical task embedding vectors in the meta-parameter pool, and to extract the corresponding model parameters as the initialization parameters of the current task. The similarity measurement method can use cosine similarity, Euclidean distance, Mahalanobis distance or other learnable measurement functions. If distance is used as a measurement, the smallest distance is selected; if similarity is used as an index, the maximum similarity is selected.

[0076] Once the target meta-parameter is determined by retrieval, it is used as the initial parameter of the current learning task model for subsequent task-specific classifier and meta-classifier initialization. This approach avoids the training fluctuations caused by random initialization, which helps to improve the convergence speed and accuracy of the model in small sample learning or task distribution transfer scenarios.

[0077] The entire process of encoding to retrieval to initialization can be designed as an end-to-end trainable substructure. During the training process, the task encoder and similarity retrieval weights are optimized simultaneously through backpropagation, so that the system has the ability of continuous learning and self-updating and optimization of retrieval effect in a task diversification environment.

[0078] A multi-layer perceptron can be used as a task encoder. The input task meta-information, such as the modality type identification vector, the sample class statistics, and the embedding mean and variance of each class of samples, are concatenated and fed into the encoder, and the task embedding vector is output. The length of the vector can be preset to 128 or 256, and can be dynamically adjusted according to the complexity of the task.

[0079] The meta-parameter pool can be implemented in a hash table structure, with each key being a task embedding vector and the value being a set of model parameters under the task. An approximate nearest neighbor (ANN) algorithm such as FAISS can be used to implement large-scale embedding retrieval, improving response efficiency. For similarity measurement, cosine similarity can be used to simplify the calculation process while maintaining good performance.

[0080] In the high-frequency task distribution switching scenario, the meta-parameter pool can also be designed as a dynamic update structure. After the training of a new task is completed, the current task embedding and the optimized model parameters are written into the parameter pool, realizing the continuous accumulation and reinforcement of meta-knowledge.

[0081] For the explainability requirement of the task embedding vector, the embedding vector can also be visualized and analyzed, for example, by projecting the embedding space using t-SNE or UMAP, so as to understand the structural relationship between different tasks at the task level.

[0082] Example: In the medical health business field, for a newly accessed disease classification task, such as switching from the "lung nodule image recognition" task to the "breast cancer ultrasound image diagnosis" task, a task embedding vector can be generated by extracting the data modality features and sample distribution features of the new task, and the existing task model parameters with similar structure can be retrieved from the meta-parameter pool to improve the training efficiency and diagnosis accuracy of the model in the breast cancer recognition task.

[0083] In the financial technology business field, when processing the task of switching from "credit card transaction fraud detection" to "cross-border fund compliance analysis", a task embedding vector can be generated based on transaction text summary and behavior time series distribution features, and the existing task parameters with the most similar semantic structure can be located from the parameter pool to realize cross-task initialization migration, shorten the model convergence period, and improve the robustness under data imbalance.

[0084] By encoding the current task and retrieving target meta-parameters from the meta-parameter pool based on the task embedding vector, the embodiment can significantly improve the initialization quality of the model on the new task, enable the model to quickly converge in a small sample scenario, improve the convergence stability and final accuracy. This structure effectively reuses the training results of historical tasks, realizes knowledge transfer and parameter sharing, reduces the performance uncertainty of the model in the early training stage, and enhances the consistency and generalization ability across tasks.

[0085] S50, establishing a task-specific classifier and a meta-classifier using the initial values of the model parameters, and updating the parameters of the task-specific classifier and the meta-classifier based on the cross-modal prototype;

[0086] In this embodiment, after the model is initialized, the construction and optimization of the classification structure are the key to achieving specific task recognition capability. On the basis of obtaining the initial value of the model parameter, two classification sub-networks with different functions but mutual cooperation are established, namely task-specific classifier and meta-classifier. The two classification sub-networks take the cross-modal prototype as the main training representation, and carry out discriminant modeling and cross-task knowledge modeling around this representation. The task-specific classifier is responsible for discriminant modeling of the sample data of the current task, and the meta-classifier is used to learn the general knowledge representation between multiple tasks and help the specific classifier to avoid overfitting and generalization error accumulation.

[0087] The task-specific classifier is a discriminant model for the current specific task, and its structure can be fully connected layer, convolution structure, graph convolution structure or Transformer structure, depending on the task modality and data structure characteristics. Its parameter initialization adopts the target meta-parameter retrieved, which ensures that the parameter space is adapted to the modality structure and sample distribution of the current task. In the training process, the classifier takes the cross-modal prototype as the input, outputs the task-specific prediction result of the corresponding label of the cross-modal prototype, and generates the task-specific loss after comparing with the corresponding real label. The loss is used to measure the modeling accuracy of the current classifier for the task target, and provides direction for subsequent parameter optimization.

[0088] The meta-classifier is used to learn the common features and general knowledge structure between tasks, and its input is mainly the cross-modal prototype, which can be combined with task embedding vectors or statistics of batch cross-modal prototypes (such as mean, variance) as auxiliary signals. The meta-classifier can be constructed in a similar way to the task-specific classifier, but its optimization target is different, and its loss function design focuses on measuring the prediction consistency between different tasks, the alignment degree of embedding representation or the stability of cross-task classification boundary. The output of the meta-classifier is often used as a regularization term or generalization constraint term to correct the tendency of the task-specific classifier to overfit the current task.

[0089] In order to improve the task modeling effect and cross-task consistency, task-specific loss and meta-knowledge loss are introduced for joint modeling. First, the task-specific loss (such as cross-entropy loss, center loss or FocalLoss) is calculated based on the cross-modal prototype and its corresponding label, and then the meta-classifier is used to model the relationship between tasks on the batch or task set composed of cross-modal prototypes, and the meta-knowledge loss (such as embedding distance loss, multi-task consistency loss or KL divergence loss) is calculated. The two are combined by weighting to obtain the joint loss function. The weight coefficient can be a preset constant, a task dynamic adjustment parameter, or generated by a control module.

[0090] On the basis of the joint loss, the model gradient is calculated by the back propagation algorithm, and the parameters of the task-specific classifier and the meta-classifier are updated synchronously. In the updating process, optimization algorithms such as Adam, SGD, RMSProp, etc. are used, and learning rate decay, weight regularization, gradient clipping and other strategies are combined to improve the training stability. This structure can complete the optimal recognition ability learning of the current task and the modeling ability training of the inter-task generality in one training cycle.

[0091] A network with a structure of two-layer linear transformation plus ReLU activation function can be selected as the task-specific classifier, and the input is the cross-modal prototype after prototype fusion, and the output is the class probability distribution. The meta-classifier is designed as a parameter-shared task discrimination network, and the input is the cross-modal prototype or its batch statistics representation, which is used to learn the discrimination boundary of the task space.

[0092] In the parameter optimization process, the following joint loss function form can be used:

[0093] L_total=L_task+λ×L_meta

[0094] Where L_task is the task-specific loss, such as softmax cross-entropy loss, L_meta is the meta-knowledge loss, such as inter-task prototype alignment loss or embedding space consistency loss, and λ is a hyperparameter that adjusts the relative weight of the two types of losses, which can be dynamically adjusted or fixed between [0, 1].

[0095] To solve the problem of unstable parameter learning when the task size changes, temperature adjustment factor, sample re-weighting mechanism and dropout strategy can be introduced to improve the generalization ability. For systems with frequent changes in task structure, structure adjustment strategies can also be used to dynamically expand or freeze part of the classifier parameters when switching between tasks to prevent catastrophic forgetting.

[0096] Example: In the medical health business field, the task-specific classifier can be used for fine-grained discrimination of image features of a certain disease, and the meta-classifier can absorb the common patterns in the image feature distribution between multiple diseases, such as similar tumor boundary performance or texture structure, to assist the model in quickly generalizing in new image tasks. During the training process, the system constructs a dedicated discriminator for each disease task, while using the meta-classifier to retain the general biological image discrimination ability.

[0097] In the field of financial technology business, when the task switches to "high-risk user detection", the task-specific classifier can focus on identifying risk behavior features in the target task, while the meta-classifier refines the common representation of user features by analyzing shared behavior patterns in historical tasks such as "credit fraud identification" or "abnormal fund flow tracking". In this way, the system can quickly adapt to new types of high-risk transactions without the need for extensive sample retraining of the model.

[0098] The embodiment realizes accurate modeling for the current task and cross-task knowledge sharing through the double-channel collaborative training mechanism based on the construction of task-specific classifiers and meta-classifiers using initialization parameters. The task-specific classifier ensures high discrimination accuracy under the current data distribution, while the meta-classifier strengthens the parameter transfer and feature generalization ability by modeling the shared structure between tasks. Both achieve collaborative optimization under the joint loss driving, which not only improves the classification performance under small sample conditions, but also reduces the risk of model overfitting caused by insufficient samples.

[0099] S60, processing the target task by the updated task-specific classifier to generate a target task processing result.

[0100] In this embodiment, after the classifier training and updating are completed, the trained task-specific classifier needs to be applied to the actual inference of specific tasks for processing the target task data to be identified and outputting the final prediction result. In this process, the multi-modal input data of the target task needs to be obtained first. The input data usually includes image, text, audio, sensor data and other different forms of information sources, depending on the task requirements. These multi-modal data have the characteristics of structural heterogeneity and semantic space difference, so the inference process needs to retain the modal collaborative modeling ability established in the training stage.

[0101] The input target task data is first structured by the defined feature extraction channel. For the image modality, the convolutional neural network model used in the training stage can be used to extract visual representation; for the text modality, a pre-trained language model (such as Transformer, BERT) can be used to extract semantic representation; for the audio modality, a spectral convolution network or RNN structure can be used to capture the time sequence features. The modal initial feature representation is generated in each channel, and is mapped to the same semantic space through dimension alignment strategy.

[0102] Then the initial modal features of the target task are input into the gating network to calculate the confidence score corresponding to each modality. The gating network structure can be a lightweight perception module such as multi-layer perceptron (MLP) or attention module, and its output is a vector of importance weight of each modality. By performing element-level multiplication between the initial modal features and the modal confidence score, dynamic weighting processing of different modalities is completed, and weighted modal features are generated.

[0103] The weighted modal features are further used to construct intra-modal prototype representations for each modality. In a specific implementation, an average pooling operation can be used to average all weighted features of the current modality in the sample dimension to generate a modal center vector. For all intra-modal prototypes, a weighted fusion is performed according to the fusion weight coefficients obtained by presetting or dynamic learning to generate a task-level cross-modal prototype representation.

[0104] The generated target task cross-modal prototype is finally input into the updated task-specific classifier. The classifier outputs the prediction result of the target task according to the input prototype vector, which can be a class distribution, a label index, a multi-label mark, or other forms of output structure, as defined by the task. In order to improve the adaptability and expression ability of the output, a post-processing mechanism can be introduced at the output end of the classifier, such as softmax normalization, multi-class confidence filtering, threshold adjustment strategy, etc., to improve the robustness of the model in unbalanced data or open label sets.

[0105] In a medical image-text joint diagnosis system, X-ray images and electronic medical record texts can be used as multi-modal inputs. Image inputs are extracted by a ResNet model, and text inputs are converted into semantic vectors by a BERT model. Then, a gating network is used to output the corresponding modal confidence scores. The initial features of the image and text modalities are weighted, intra-modal prototypes are calculated, and a cross-modal prototype is obtained by linear fusion. The prototype vector is input into a task-specific classifier, and the output is a disease prediction label (such as whether or not to have pneumonia). The classifier output can be further filtered by a confidence threshold mechanism to enhance the stability and safety of the output.

[0106] In a financial transaction behavior recognition system, structured transaction data and unstructured user description texts are used as inputs. The transaction data is extracted by a multi-layer perception network, and the text is extracted by a Transformer model. The above features are weighted and fused by a gating network to generate a cross-modal transaction prototype, which is input into an updated task-specific classifier. The classification result can be a user risk level, a fraud behavior label, etc. The post-processing module can provide a high-confidence prompt or rule-enhanced verification for abnormal samples.

[0107] By applying the updated task-specific classifier to process new target task data, the model can be quickly migrated from the training task to the application task, ensuring high-precision inference ability under small sample conditions. The weighting mechanism between modalities effectively reduces the interference of low-quality modalities, and the cross-modal prototype generation mechanism ensures the consistency of multi-modal collaborative representation, so that the classifier can stably discriminate and output the results of the target task.

[0108] The application relates to the technical field of artificial intelligence, can be applied to business scenes such as financial technology and medical health, and discloses a data processing method and device based on adaptive meta-learning, equipment and a medium, which comprises the following steps: acquiring multi-modal input data and extracting initial modal features, generating modal confidence scores by using a gating network and performing weighted processing on the initial modal features to obtain weighted modal features; generating intra-modal prototypes according to the weighted modal features, performing weighted fusion on the intra-modal prototypes by using a fusion weight coefficient to obtain cross-modal prototypes; encoding a current learning task to obtain a task embedding vector, and retrieving target meta-parameters from a meta-parameter pool to initialize model parameters; constructing a task-specific classifier and a meta-classifier based on the initialized parameters and updating parameters based on the cross-modal prototypes, processing a target task by using the updated task-specific classifier, and outputting a final processing result. By introducing the modal confidence score in the feature extraction stage, the application realizes dynamic weighting of the multi-modal input data, effectively reduces modal noise interference, introduces the fusion weight coefficient in the prototype generation and fusion stage, enhances the adaptation ability between modes, encodes the learning task and retrieves the meta-parameters of the most similar task as the initialization, significantly improves the rapid adaptation ability and convergence speed of the model under a new task, and thus realizes higher task generalization performance and robustness in a multi-modal small sample scene.

[0109] In one embodiment, the above step S10 comprises:

[0110] S101, receiving original input data containing image modal data, text modal data and audio modal data;

[0111] S102, assigning the image modal data to an image processing channel, and obtaining image initial modal features by processing through a convolution feature extractor;

[0112] S103, assigning the text modal data to a text processing channel, and obtaining text initial modal features by processing through a sequence feature extractor;

[0113] S104, assigning the audio modal data to an audio processing channel, and obtaining audio initial modal features by processing through a spectrum feature extractor;

[0114] S105, performing normalization processing on the image initial modal features, the text initial modal features and the audio initial modal features;

[0115] S106, unifying the normalized image initial modal features, the normalized text initial modal features and the normalized audio initial modal features to the same dimensional space to form the initial modal features.

[0116] In this embodiment, when facing the problem of modeling tasks while participating in heterogeneous modal information, a stable and modal adaptation capable data input path must be first constructed to complete the unified mapping process of multi-modal raw data to the structured feature space. Multi-modal input data contains three types of raw information: images, texts, and audios. Each modality has essential differences in representation structure, collection source, semantic dimension, etc., and cannot be directly used for unified modeling, so modal decoupling and feature extraction are needed first.

[0117] The received multi-modal data should have data structure divisibility, that is, when designing the input interface, an image input channel, a text input channel, and an audio input channel are established respectively to support identifying the original data stream by modality type and routing it to the corresponding processing path. Image modal data can include medical images, industrial photos, face images, etc. two-dimensional pixel data structure, usually represented by RGB matrix or grayscale matrix. Text modal data includes natural language sentences, label descriptions, user feedback, etc. usually as character or word vector sequence. Audio modal data such as speech signals, environmental sounds, etc. are in the form of waveform data and need to be preprocessed and converted into spectral representation for downstream processing.

[0118] In the image processing channel, the input image data is processed by a convolution feature extractor, which can be a pre-trained or self-defined convolutional neural network structure such as ResNet, DenseNet, MobileNet, etc. The structure extracts multi-level spatial semantic features through consecutive convolution, pooling, and nonlinear activation operations, and outputs image initial modal features in tensor form. The extractor should have strong local perception ability to maintain the spatial continuity of key structures in the image modality.

[0119] In the text processing channel, the input text modal data is processed by a sequence feature extractor, which can be based on recurrent neural network (RNN, LSTM), Transformer encoder, BERT, etc. deep language modeling network. Through the word embedding matrix, the discrete character or word sequence is mapped to a dense vector sequence, and then the text semantic vector is generated through time-dependent modeling or attention mechanism as the initial modal feature of the text. The extractor should have the ability to capture context dependence and semantic combination.

[0120] In the audio processing channel, the audio raw signal is input into the spectral feature extractor after being transformed into a frequency spectrum by short-time Fourier transform or Mel spectrum analysis. The extractor usually uses a two-dimensional convolution network or spectral convolution block to extract local time-frequency features, and outputs audio initial modal features reflecting tone, rhythm, energy change, etc. The feature extraction module should have the ability to analyze time series information and frequency change patterns to improve the accuracy of describing non-stationary patterns in audio.

[0121] The modal initial features output by the three channels are in different scale ranges, distribution structures, and vector dimensions. To ensure the feasibility of subsequent feature fusion and cross-modal modeling, normalization processing is required for each type of initial modal feature. The normalization method can use Z-score standardization, min-max normalization, or batch normalization strategy to make the modal feature vectors comparable in numerical scale, avoiding model bias caused by a large numerical scale of a certain modal.

[0122] After normalization, the image initial modal feature, the text initial modal feature, and the audio initial modal feature need to be aligned in a unified dimension. This processing step is completed by a dimension mapping network, which projects each modal feature into a unified vector space. A fully connected projection layer or a shared dimension transformation module can be used to map all modal features into vectors of the same dimension size. In this process, modal labels or modal embeddings can be introduced to guide the semantic alignment of the unified dimension space.

[0123] Finally, the image, text, and audio initial features are unified into an initial modal feature set with equal dimensions, standardized value ranges, and comparable semantics, laying a foundation for subsequent dynamic weighting modeling, modal prototype construction, and cross-modal fusion.

[0124] This embodiment can effectively solve the modeling obstacles caused by the heterogeneity of multi-modal data by constructing image, text, and audio input channels and extracting high-level semantic representations using a structured feature extractor. The introduction of normalization mechanism and unified dimension mapping strategy can ensure the consistency of different modal features in numerical scale and vector structure, thereby significantly improving the subsequent fusion quality and feature expression ability. This mechanism does not rely on a fixed modal feature concatenation order and has stronger generalization and adaptation ability. For tasks with significant modal differences, it can avoid the interference of modal imbalance on the final decision-making process and effectively unify modal collaborative modeling, providing a more stable and discriminative input basis for subsequent model learning.

[0125] In one embodiment, the above step S20 includes:

[0126] S201, for each modal initial modal feature, applying a weight matrix corresponding to the modal to perform linear transformation to obtain linearly transformed features;

[0127] S202, adding a bias term corresponding to the modal to the linearly transformed features to obtain bias-adjusted features;

[0128] S203, applying a nonlinear activation function to the bias-adjusted features to generate activated features;

[0129] S204, applying a normalization function to the activated features to generate the modal confidence score;

[0130] S205, multiplying the initial modality feature of each modality with the corresponding modality confidence score to obtain a weighted feature of the corresponding modality;

[0131] S206, combining the weighted features of all modalities to obtain the weighted modality feature.

[0132] In this embodiment, in the fusion processing of multi-modal data, different modalities usually have non-uniformity in semantic expression ability, information density and task relevance. If an equal weight strategy is used in the modeling process, it is easy to cause noise transmission of invalid modalities or weak modalities, which has a negative impact on the overall model performance. Therefore, after the uniform processing of input modality features, a dynamic weighting mechanism needs to be introduced to proportionally regulate the feature expression according to the importance of each modality in the task, so as to generate weighted modality features that are unified in structure but have differences in semantics.

[0133] This process is based on internal information guidance of the initial modality feature to construct a gating network module with input perception ability. The gating network receives the initial modality feature input of each modality, and constructs a set of learnable weight matrix parameters for each modality. The setting of these weight matrices is not fixed, but consistent with the feature dimension of the modality, which is used to perform linear transformation operation to project the original feature from the original space to a new embedding space. This transformation does not change the structural properties of the modality, but can adjust the feature direction and expression strength to provide a basis for subsequent weight generation.

[0134] After completing the linear transformation, the transformed feature is added to a set of preset or trained bias terms to form a bias-adjusted feature. The purpose of introducing the bias term is to break the center contraction phenomenon that may be caused by pure linear transformation, so that the features of each modality have higher freedom in the range of change. This addition operation does not destroy the structural relationship between the original features, but can enhance the asymmetric distribution adaptation ability of each modality feature.

[0135] Subsequently, the bias-adjusted feature is input into a nonlinear activation function module, such as ReLU, GELU, Sigmoid or Swish, to enhance the expression ability. The introduction of the activation function enables the model to have nonlinear segmentation ability when processing the feature, which can automatically focus on the key modality feature dimension while suppressing the expression of redundant or low-correlation features. This nonlinear mapping process does not involve cross-coupling between modalities, ensuring that each modality is independently weighted.

[0136] To facilitate the comparability and normalization of inter-modal confidence, a normalization function is applied on the basis of the activation features. The Softmax function, normalized L2 norm or BatchNorm method can be used to convert the modal activation results into modal confidence scores. The modal confidence score is a scalar or vector representation of the importance of the modal, and its numerical range is usually controlled between 0 and 1, representing the relative contribution of the current modal feature to the overall task goal.

[0137] Subsequently, the modal confidence score is multiplied by the initial modal feature of the corresponding modal to generate a weighted feature. The multiplication operation is performed at the tensor level, supporting the broadcast mechanism, that is, whether the confidence is a scalar or a vector, it can be multiplied by the modal feature element by element to realize the dynamic adjustment of the feature in the semantic expression intensity. This process does not introduce new information, but only adjusts the expression proportion of the original feature based on the confidence.

[0138] After all the modalities complete independent weighting processing, all the weighted modal features are combined in the specified order. The combination operation can use vector splicing, weighted summation or stacking operation, and is flexibly adjusted according to the specific requirements of the downstream task for the fusion form, and finally generates a weighted modal feature with a unified expression structure. The weighted modal feature has adaptive weight adjustment capability on the basis of maintaining the original modal semantics, providing a more semantically distinguishable input structure for subsequent modal prototype generation and task modeling.

[0139] The embodiment can generate modal confidence scores with distinguishability and adaptability according to the information expressed by each modal feature itself by constructing a modal weight generation path based on a gating mechanism, effectively improving the modal adaptation capability of the model in the multi-modal task. The layer-by-layer processing of linear transformation, bias adjustment, activation mapping and normalization ensures the stability and consistency of the modal weight in the representation ability and numerical distribution, avoiding the problem that the weights between modalities cannot be compared. Finally, through the element-by-element weighting operation of the confidence score and the initial feature, the weak modal feature is suppressed and the strong modal feature is amplified, thereby improving the contribution of the entire input feature to the downstream task.

[0140] In one embodiment, the above step S30 comprises:

[0141] S301, obtaining the weighted modal feature of each modality;

[0142] S302, determining the average value of the weighted modal feature of each modality to generate an intra-modal prototype of the corresponding modality;

[0143] S303, dynamically generating a fusion weight coefficient of each intra-modal prototype through a meta-learning mechanism;

[0144] S304, multiplying the intra-modal prototype of each modality by the corresponding fusion weight coefficient to obtain a weighted intra-modal prototype;

[0145] S305, performing weighted summation on all the weighted intra-modal prototypes to obtain the cross-modal prototype.

[0146] In the present embodiment, in the multi-modal task, the distribution of each modality expression in the semantic space is heterogeneous, and direct fusion is easy to cause semantic conflict or imbalance of modality information. Therefore, it is necessary to construct an internal aggregated representation for each modality before inter-modal fusion, and to weight and regulate each modality based on the task state at the cross-modal level, so as to generate a unified task representation. The key in this process is to capture the common structure within the modality through the intra-modal prototype, and to realize the adaptive joint expression between different modalities through the fusion weight.

[0147] The intra-modal prototype refers to the semantic center expression formed after the aggregation of multiple samples or features in the same modality in multi-modal learning. It is usually in the form of the average value of the feature vectors of all samples in the modality, representing the common semantic information of the modality under the current task or category. Its essence is a low-dimensional dense expression, which can compress the information redundancy within the modality while retaining the discriminative features specific to the modality. This expression form not only has strong representativeness, but also can effectively support subsequent matching, fusion and reasoning operations with other modality features. In practical applications, the intra-modal prototype is usually obtained by calculating the weighted feature mean of all samples in each modality. The weight can vary according to the modality confidence, sample importance or task context, so that the prototype expression is more suitable for the current task semantic requirements. Through the introduction of the intra-modal prototype, the internal rules of each modality can be fully explored before fusion, improving the consistency and robustness of the fusion between different modalities.

[0148] Obtaining weighted modality features for each modality is a prerequisite for constructing intra-modal prototypes. The weighted modality features have been regulated by confidence, and they contain the importance of each modality in semantic representation in the current task. At this time, the independence of the modality is still retained, and no fusion operation is performed. Therefore, the weighted modality features are grouped according to the modality dimension, so that each group only contains feature sets belonging to the same modality.

[0149] In the feature set of each modality, in order to obtain the center representation of the modality in the current task semantic space, the mean value of the weighted modality features of all samples in the modality is calculated. This operation is equivalent to finding the center point of the modality features in the Euclidean space, thereby forming the intra-modal prototype. The mean value calculation process can be realized by summing up each dimension and dividing by the number of samples. The intra-modal prototype has compression characteristics, which projects redundant individual features to a centralized expression vector, facilitating subsequent unified modeling.

[0150] Although the intra-modal prototypes are representative expressions of each modality, different tasks may have different dependencies on modalities. For example, in the image recognition task, the intra-modal prototype of the image modality should be dominant; while in the text generation task, the text modality prototype should bear the core information. Therefore, it is necessary to assign a fusion weight coefficient to each intra-modal prototype to reflect the importance of the modality in the current task.

[0151] The fusion weight coefficient should not be a static parameter, but should have task perception ability, so it is dynamically generated by using a meta-learning mechanism. The meta-learning mechanism receives input information such as task embedding and modality feature distribution of the current task, and trains a generalizable weight prediction module. The module outputs a set of fusion weight coefficients consistent with the number of modalities, with a value range of 0 to 1 and a normalization constraint, so that multiple modal prototypes can be weighted and fused in the equivalent space.

[0152] The intra-modal prototype of each modality is multiplied element by element with its corresponding fusion weight coefficient to obtain the weighted intra-modal prototype. This operation is essentially a re-scaling of the spatial expression ability of the modal prototype, and encodes task perception information into the representation of the modal layer. Each modal prototype is mapped to a fusion state, with stronger context adaptation ability.

[0153] All weighted intra-modal prototypes are weighted and summed to form a cross-modal prototype. This operation is equivalent to weighted averaging in tensor dimensions, and has the effect of modeling multiple modal semantics uniformly. The cross-modal prototype not only integrates the compressed representations of each modality, but also implicitly reflects the dependency structure of the current task on the modal, which is a high semantic concentration representation for subsequent classification, prediction or generation.

[0154] The above construction process extends from independent processing of the modal layer to unified modeling of the fusion layer, ensuring that semantic information is aggregated step by step, while avoiding the risk of modal conflict caused by early fusion. By dynamically generating fusion weights, a task state regulation mechanism is introduced, making the cross-modal prototype have stronger task-specific expression ability.

[0155] The embodiment realizes progressive structural conversion from intra-modal feature aggregation to cross-modal semantic fusion by introducing a two-stage regulation mechanism of intra-modal prototype and fusion weight coefficient. The mean operation captures the commonality of the modal, and the fusion weight introduces task adaptability, effectively solving the information conflict problem caused by modal heterogeneity in multi-modal tasks. Compared with the direct splicing or attention fusion method, this method has better stability and controllability. In the task of incomplete or uneven quality of modal, such as the risk prediction task driven by financial time series data and text news, the interference of low-quality modal can be effectively suppressed; in the medical multi-modal diagnosis, when the image resolution fluctuates or the text description is redundant, the modal weight can also be dynamically regulated to improve the accuracy and robustness of diagnosis. The construction of cross-modal prototype takes into account the modal aggregation and task adaptability, and constitutes a unified input for downstream modeling, which helps to improve the generalization ability and convergence efficiency of the final model in multi-modal small sample tasks.

[0156] In one embodiment, the above step S40 comprises:

[0157] S401, obtaining task feature data of a current learning task;

[0158] S402, extracting core feature representation of the task feature data;

[0159] S403, processing the core feature representation by a task encoder to generate a task embedding vector;

[0160] S404, obtaining a plurality of historical task embedding vectors in a meta-parameter pool;

[0161] S405, determining the similarity of the task embedding vector and each historical task embedding vector;

[0162] S406, selecting a meta-parameter corresponding to the highest similarity as a target meta-parameter;

[0163] S407, taking the target meta-parameter as an initial value of the model parameter.

[0164] In the embodiment, the task feature data of the current learning task can be derived from task description, sample label distribution, data statistical information or feature summary generated in the previous stage. By collecting these highly relevant structural or non-structural information of the current task, the overall representation basis of the task is constructed. Such data usually reflects the type of task, input modal combination, data noise characteristics, class structure, sample density and other elements, and has high task discrimination ability.

[0165] The process of extracting core feature representation from task feature data relies on a specific set of data encoding and abstraction mechanisms. In the implementation process, multi-layer perceptron, Transformer encoder, or dual-tower encoder structure can be used to represent the input features, filter out invalid information, and retain semantic vectors that can distinguish task attributes. The final generated feature representation should have low dimensionality, compactness, strong measurability, and high relevance to historical task distribution.

[0166] Further processing of the core feature representation by the task encoder can output a task embedding vector. This vector is located in a unified embedding space with fixed dimensions and good geometric distribution characteristics. The task encoder can use projection network, attention mechanism network, or graph neural network structure, and can be flexibly selected according to the actual task structure. This process maps the current task semantics to a space coexisting with historical task embeddings, providing a unified measure for subsequent similarity retrieval.

[0167] The meta-parameter pool stores a large number of optimal parameter sets of historical tasks and corresponding task embedding vectors. These meta-parameters represent the optimal model structure or weight configuration obtained by training the model on historical tasks, and are constructed through offline clustering or online aggregation. Each meta-parameter corresponds to an embedding vector, which together form a parameter-task mapping index structure.

[0168] By measuring the similarity between the current task embedding vector and each historical task embedding vector, the most matching historical task can be retrieved. The similarity measurement method is usually cosine distance, Euclidean distance, or kernel function-based measurement mechanism to ensure that the most similar task is captured. When performing efficient retrieval, KD-Tree, ball tree, or approximate nearest neighbor search algorithm can be used to improve query speed.

[0169] Select the meta-parameter corresponding to the historical task with the maximum similarity as the initialization model parameter for the current task. These parameters can be weight matrices, bias terms, BatchNorm statistics of neural networks, or more abstract network structure configurations such as activation function combinations, module connection methods, etc. This initialization parameter will be directly used to construct downstream classifier and discriminator structures, allowing the model to converge faster.

[0170] The embodiment encodes the current task to obtain a measurable task embedding representation, and retrieves the most similar historical task parameters in the meta-parameter pool as initialization values based on the representation, which can significantly improve the learning efficiency of the model in small sample tasks. Compared with the traditional random initialization method, the method can reuse the knowledge learned from historical tasks, provide semantic priori for the representation of the current task, reduce the gradient fluctuation amplitude, and improve the convergence speed of model parameter adjustment. At the same time, since the parameter initialization is more close to the semantic structure of the target task, the generalization ability of the model is also enhanced. In the task frequent switching or cross-modal task migration scene, the mechanism effectively reduces the sample demand and parameter adjustment cost, and realizes efficient learning under the condition of small sample.

[0171] In one embodiment, the above step S50 comprises:

[0172] S501, initializing a task-specific classifier and a meta-classifier using the initial value of the model parameter;

[0173] S502, processing the cross-modal prototype through the task-specific classifier to obtain a task-specific prediction result;

[0174] S503, determining a task-specific loss according to the task-specific prediction result and a real label corresponding to the cross-modal prototype;

[0175] S504, obtaining a meta-knowledge analysis result by analyzing the common mode between tasks based on the cross-modal prototype through the meta-classifier;

[0176] S505, determining a meta-knowledge loss according to the meta-knowledge analysis result;

[0177] S506, obtaining a meta-knowledge loss weight coefficient, and multiplying the meta-knowledge loss by the meta-knowledge loss weight coefficient to obtain a weighted meta-knowledge loss;

[0178] S507, adding the task-specific loss and the weighted meta-knowledge loss to generate a total loss;

[0179] S508, determining a gradient according to the total loss;

[0180] S509, updating the parameters of the task-specific classifier and the meta-classifier through back propagation based on the gradient.

[0181] In the embodiment, first, the initial value of the model parameter is used to initialize two independent structures: a task-specific classifier and a meta-classifier. The initial value of the model parameter comes from the set of optimal historical task parameters retrieved through task embedding vector matching in the meta-parameter pool, which can provide the most similar model starting state to the current task to enhance the effectiveness of the model in the early stage of training.

[0182] The task-specific classifier is responsible for specific classification of the current task sample, and its structure can adopt a multi-layer perceptron or a Transformer-based sub-module, which is adjusted according to the input sample dimension and task type. The classifier conducts discriminant modeling with the cross-modal prototype as the unified input representation. The training data input into the classifier produces a prediction output, which is then compared with the actual label to obtain the task-specific loss. The task-specific loss is generally measured by loss functions such as cross-entropy and mean square error, reflecting the classification performance of the current task model.

[0183] The meta-classifier, as the execution subject of the parameter constraint mechanism, does not directly make predictions, but learns common knowledge between tasks, such as shared representation structures and transfer patterns. The meta-classifier takes the cross-modal prototype as the main input source, and can also use task embedding vectors or batch cross-modal prototype statistics (mean, variance, covariance) as auxiliary signals. The analysis of common patterns between tasks relies on the structure and function design of the meta-classifier. This classifier does not directly participate in the prediction process of specific tasks, but is used to model the potential feature structures and generalization rules shared by multiple tasks. There may be significant differences in data modalities, objectives, or sample sizes between tasks, and parameters learned directly from a single task often have high specificity and lack transferability. To extract structural knowledge that can be shared and transferred by multiple tasks, a meta-model that is optimized in multiple task experiences is needed to extract stable representation space distribution or gradient direction, i.e., meta-knowledge.

[0184] The input of the meta-classifier can be an abstract representation of multiple task samples, such as intermediate features generated by an encoder or task embedding, or the gradient response of each task loss to a specific parameter, the embedding expression of the optimization path trajectory or the model parameter state. In the multi-task parallel training or playback training scenario, the cross-modal prototypes of multiple tasks are input into the meta-classifier for joint encoding and contrastive learning to characterize the structural association and similarity between tasks. For example, the cross-modal prototype sequence can be used directly instead of weighted modal features to identify recurring distribution structures or modal preferences in different tasks, thereby identifying common discriminant directions or weight tendencies.

[0185] In terms of structure design, the meta-classifier can use a graph neural network to construct a similarity graph between tasks, use a shared attention mechanism to strengthen cross-task interaction learning, or use a dual-tower structure to separate the representations of source and target tasks and perform matching operations in a high-level space. Through these mechanisms, the meta-classifier forms the ability to express "which features are shared between tasks and which are transferable."

[0186] The meta-knowledge analysis result is the output of the above analysis process, usually represented in the form of a vector structure, a strategy matrix, a parameter adjustment direction, or a task relationship label, which can be used to further adjust the parameter update strategy of the task-specific classifier, constrain the weight adjustment of the loss function, or select the representation when initializing a new task model. When the input is a cross-modal prototype, the meta-knowledge analysis result can also be embodied as the subspace alignment direction of the cross-modal prototype, the edge weight of the task similarity graph, or the regularization term coefficient. The essence of this result is the abstract modeling of the structural similarity between tasks, which is a powerful extension of the "task irrelevance" assumption in traditional meta-learning.

[0187] In one class of embodiments, the cross-modal prototypes corresponding to different tasks are taken as the input of the meta-classifier, processed through a shared encoding network, and a feature correlation matrix between tasks is output. This matrix is used to identify the structural similarity between the current task and the historical task, thereby assisting the training optimization direction of the current task. For example, in a graph neural network architecture, each task can be regarded as a node in the graph, and the weight of the edge between the nodes represents the similarity between the tasks, and the features of the edge can come from the statistical representation of the cross-modal prototype or the task embedding. The meta-classifier propagates the structural knowledge through graph convolution operation to achieve shared learning.

[0188] In another class of embodiments, a task comparison module is constructed to compare the feature representations between each pair of tasks, where the feature representations to be compared are taken from the cross-modal prototypes or their linear mappings, and a multi-task contrast loss function is used to optimize the parameters of the meta-classifier. Through this module, the meta-classifier learns how to structure cluster or layer the tasks in the feature space, and outputs a general representation that can be used for subsequent transfer, such as a general attention weight or a parameter sharing matrix. These outputs serve as meta-knowledge analysis results, which are passed back to the task-specific classifier to adjust its representation learning strategy.

[0189] In multi-modal tasks, the intra-modal prototypes in different modalities can continue to be used as auxiliary input, or the cross-modal prototypes can be directly used as the only input, and the meta-classifier can be used to model the coordination mechanism of the inter-modal weights in different tasks. The meta-knowledge analysis result in this case can be embodied as a cross-modal weight pattern or a modal confidence transfer rule, which is used to guide the importance allocation of each modality in the target task.

[0190] Subsequently, in order to construct an effective loss function, the above two losses are combined by weighting. The task-specific loss is calculated based on the cross-modal prototype and the corresponding label, and the meta-knowledge loss is calculated based on the relationship modeling result of the cross-modal prototype batch or the task set. In this process, a learnable or preset weight coefficient is introduced to balance the influence proportion of the task-specific loss and the meta-knowledge loss, so as to obtain the total loss. The weight coefficient can be dynamically adjusted according to the task complexity, or obtained through hyperparameter tuning.

[0191] The gradient is calculated according to the total loss, by solving the partial derivative of the loss function with respect to each network parameter. The process can be automatically completed by the back propagation algorithm, the key of which is to make the gradient signal derived from the cross-modal prototype and the meta-knowledge regular term jointly act on the parameter update path, so as to realize the synchronous optimization of the parameters of the two networks.

[0192] The whole process ensures the unity of task specificity and task generality, enables the classifier to adapt to the current task faster and transfer effective experience from previous tasks under the condition of limited sample size, and reduces the training bottleneck caused by the dependence on large-scale labeled data.

[0193] In this embodiment, the task embedding vector guided model parameter initialization is combined with the double-channel classifier structure, and the cross-modal prototype is used as a consistent input representation for training and constraint. In the case of large differences between tasks or extremely small samples, the structure can quickly learn the discriminative features of the current task in the task-specific classifier, and at the same time, the meta-classifier can capture the structural commonality between tasks to provide continuous learning supervision signals for the model. By combining the two loss sources with weights, the dynamic balance between individualization and generalization ability is realized, so as to improve the adaptation speed and final accuracy of the model to new tasks, significantly reduce the training sample requirement, and enhance the generalization ability in the small sample scenario.

[0194] In one embodiment, the above step S60 comprises:

[0195] S601, obtaining multi-modal input data of a target task;

[0196] S602, performing feature extraction on the multi-modal input data of the target task to obtain initial modal features of the target task;

[0197] S603, generating a target task modal confidence score based on the initial modal features of the target task through a gating network, and performing weighted processing on the initial modal features of the target task based on the target task modal confidence score to obtain weighted modal features of the target task;

[0198] S604, determining the average value of the weighted modal features of the target task of each modality to generate a target task intra-modal prototype of the corresponding modality;

[0199] S605, weighting and fusing the target task intra-modal prototypes of the respective modalities based on a fusion weight coefficient to generate a target task cross-modal prototype;

[0200] S606, inputting the target task cross-modal prototype into an updated task-specific classifier to obtain a target task prediction result;

[0201] S607, decoding the target task prediction result to output the target task processing result.

[0202] In this embodiment, the processing process of the target task starts with obtaining its corresponding multi-modal input data, which can contain different information dimensions such as images, texts, audios, etc. Each modality has independent structural characteristics and expression domain. In order to ensure that each modality input can be unified into the learning process of the subsequent model, differentiated data acquisition interface and data preprocessing mechanism need to be used. Image modal data can come from sensor image frames, diagnostic images, security monitoring screenshots, etc. Text modal data includes natural language description, medical record text, financial semantic label, etc. Audio modal data can cover heart sound, voice command, market call record, etc.

[0203] After obtaining the multi-modal input data of the target task, the features of each modality need to be extracted synchronously and adaptively to the heterogeneous structure. Image modal usually extracts spatial structure information through convolutional neural network, such as using ResNet, EfficientNet, etc. Text modal can capture context semantics through Transformer structure such as BERT, RoBERTa, and audio modal can extract frequency domain signal features through one-dimensional or two-dimensional convolution network after spectrum conversion. In this stage, the timing or semantic alignment structure of each modality needs to be maintained, and the mapping dimension of the final extracted modal feature needs to be consistent or can be mapped.

[0204] The extracted initial modal features of the target task still have the problem of inter-modal imbalance, so feature weighting needs to be performed through a gating network. The gating network receives the initial modal features of each modality as input, linearly maps the input features through a set of learnable parameter matrices, adds a bias item and inputs an activation function to generate a nonlinear response, and then maps each modality output to the [0, 1] interval through normalization operation as a modal confidence score, representing the contribution weight of this modality to the prediction result in the current task. The modal confidence score and its corresponding initial modal feature are multiplied element by element to form the weighted modal feature of the target task. Weighting processing not only enhances the representation strength of high-confidence modal in the numerical level, but also guides the modal attention distribution in the optimization space when aggregating the prototype later.

[0205] To further aggregate the inter-modal information, the weighted modal features of each modality are respectively subjected to a mean aggregation operation in the feature dimension, generating the intra-modal prototypes of the target task. This process is a kind of structural compression of the internal representation of each modality, and the obtained intra-modal prototypes reflect the central semantics or decision focus of the modality in the current task context. After the generation of multiple intra-modal prototypes, fusion weight coefficients need to be introduced to form the final unified representation structure. These coefficients can be statically set or dynamically generated by other network modules, and are used to regulate the proportion of different modalities in the final cross-modal representation. The weighted sum of all intra-modal prototypes and their corresponding fusion weight coefficients yields the cross-modal prototype of the target task.

[0206] The cross-modal prototype of the target task is regarded as a unified expression form after fusing the multi-modal context information, and it is input into the task-specific classifier that has been updated. This classifier has adjusted its parameter structure to adapt to the characteristics of the current task through joint training with the meta-classifier, total loss backpropagation, and other processes during the pre-training stage, so it can perform high-robustness discrimination on the current cross-modal prototype and output structured prediction results of the target task, such as label probability distribution, classification label index, or sequence output.

[0207] To convert the model output into an interpretable result that can be used in specific application scenarios, decoding processing needs to be performed on the prediction results. The decoding process may include argmax operation, label mapping, output logic conversion, etc. depending on the task type. For example, in a classification task, the output result is matched with the business label set after selecting the highest confidence class label through argmax operation; in a sequence task, beam search decoding or CTC decoding needs to be performed. Finally, structured target task processing results are formed, such as financial fraud judgment labels, disease risk levels, or multi-modal event recognition results, etc.

[0208] Example: In the task of clinical auxiliary diagnosis, doctors need to integrate the patient's medical images, electronic medical record texts, and voice medical history records to judge the possible disease categories or health risk levels of the patient. To realize an intelligent multi-modal disease recognition system, first, collect the patient's related multi-modal input data, including image modalities such as chest X-ray, CT images, MRI images, text modalities such as structured and unstructured electronic medical record texts, doctor's handwritten medical history records, and test sheet data, and audio modalities from voice communication or voice medical history statements during doctor-patient consultations. These data are collected through medical institution HIS (Hospital Information System), PACS (Picture Archiving and Communication System), and voice transcription interfaces.

[0209] The image data is input into a pre-trained convolutional neural network structure (such as ResNet50) to extract image modal features; the text data is input into a BERT or BiLSTM model to extract semantic embedding representations; and the audio signal is input into a CNN or 1D-Transformer structure after Mel spectrum conversion to extract frequency domain features. In order to make the three types of modal features comparable and fusible, normalization processing is performed on them, and they are adjusted to the same vector dimension space through full connection mapping, thereby forming standardized initial modal features.

[0210] To solve the problem of different modal contributions in different cases, a gating network is used to perform adaptive weighting processing on each modal. The image modal may have a very high weight in the diagnosis of lung diseases, while in neurological diseases such as depression or Parkinson's syndrome, the audio and text modalities are more valuable for diagnosis. The gating network generates modal confidence after linear transformation, bias adjustment, and activation normalization of the initial features of each modal, and then guides the weighting processing of the modal features. The weighted modal features formed after fusion retain the feature expressions of different modalities and dynamically allocate modal weights according to the current condition.

[0211] Subsequently, the weighted feature mean of each modal is calculated to generate an image modal prototype, a text modal prototype, and an audio modal prototype. Taking chronic obstructive pulmonary disease (COPD) as an example, the image prototype exhibits bronchial wall thickening and emphysema characteristics, the text prototype may focus on "chronic cough" and "pulmonary function decline" word embeddings, and the audio prototype captures features such as shortness of breath and coughing sounds. On this basis, a meta-learning mechanism is used to dynamically generate fusion weight coefficients, which determine which modalities should be more critical in the final judgment based on disease type, patient information, and task context. For example, in the judgment of early Alzheimer's disease, the system may give higher fusion weights to audio and text. The intra-modal prototype after fusion weighting becomes a cross-modal prototype vector, which serves as the patient's comprehensive representation.

[0212] The system encodes the task metadata according to the current task (such as disease classification, disease stage judgment, etc.), such as task type (disease recognition / disease classification), data dimension (whether containing time series), label dimension (multi-class / multi-label), etc., which are input into the task encoder to generate a task embedding vector. The highest similarity between the historical task embedding is retrieved in the preset task history knowledge base to obtain the corresponding meta-parameter as the initialization parameter, thereby establishing a task-specific classifier and a meta-classifier. The task-specific classifier is used to predict the disease type of the current patient, while the meta-classifier is used to identify potential common medical patterns among different tasks, such as similar image manifestations among respiratory diseases.

[0213] During the training process, the task-specific classifier outputs the predicted label (e.g., whether it is a lung nodule, benign or malignant nodule, etc.) according to the training sample, and the loss is calculated with the real label; the meta-classifier analyzes the common characteristics of multiple cases and outputs the judgment result of the common law, and further calculates the meta-knowledge loss. The system obtains the meta-knowledge loss weight coefficient under the current training round, weights the meta-knowledge loss, and combines it into the task-specific loss to form the total loss and calculate the gradient, and update the classifier parameters through back propagation.

[0214] In actual deployment, when a new patient case is input into the system, the system will repeat the above data acquisition, feature extraction, weighting processing, prototype generation and fusion process, input the cross-modal prototype generated by it into the trained task-specific classifier, output the disease prediction result (such as disease classification label, risk level), and output it as structured diagnostic information after decoding processing, such as “moderate chronic obstructive pulmonary disease, suggest rechecking CT, transfer to respiratory medicine department”.

[0215] In the intelligent risk control system, it is necessary to integrate multi-source heterogeneous information from customers to achieve accurate credit evaluation and fraud detection tasks. The input data includes structured transaction data (such as transaction records, transfer behavior sequences), text data (such as loan application descriptions, customer service interaction records), and audio data (such as customer voice emotion and tone characteristics in phone recordings), and the system needs to perform deep discriminant modeling based on multi-modal information to deal with complex financial behavior patterns and highly unbalanced small sample risk control tasks.

[0216] The system first receives the above multi-modal raw input and processes the modalities through three types of special channels respectively. The structured transaction data is sent to a multi-layer perception or graph neural network to extract the graph embedding representation of the time series behavior; the text data is input into a pre-trained language model (such as RoBERTa) to extract the context semantic features; the audio data is converted into a mel spectrum graph and sent to a residual convolutional network to extract the frequency spectrum energy change features. Subsequently, all modal features are normalized to the same feature space dimension to form initial modal features, ensuring vector consistency and composability between cross-modalities.

[0217] To dynamically identify the contribution of each type of modality in the current risk control task, a gating network is introduced to perform adaptive weighting processing. For example, in the “fraud detection” task, transaction behavior patterns are usually more critical, while in the “customer credit judgment”, the promise language contained in the customer's voice emotion or text description is more discriminative. The gating network performs weight transformation, bias adjustment, activation processing and normalization on each initial modality feature to generate modality confidence scores. The initial features of each modality are weighted element by element with the corresponding confidence scores, and the weighted modality features of each modality are output and combined to form a comprehensive representation.

[0218] Subsequently, the system calculates the mean vector for the weighted features of each modality, respectively constructing the transaction modality prototype, the text modality prototype, and the speech modality prototype. For example, for a credit scoring task of a certain customer group, the transaction modality prototype can be expressed as the vector center of high-frequency micro-transaction, the text modality prototype focuses on the key expressions in the financial status description, and the speech modality prototype reflects the language smoothness and emotion indicators. By dynamically adjusting the contribution proportion of each modality prototype through the fusion weight coefficient, the modality effectiveness difference under different task backgrounds is addressed. Finally, a cross-modality prototype vector representing the overall financial behavior characteristics of the customer is output through weighted fusion.

[0219] Next, the system performs structured coding on the current financial task. The task coding includes task category (such as "risk grading" or "fraud identification"), label granularity, modality combination structure, etc., and obtains the task embedding vector after embedding layer processing. The system retrieves the embedding representation of historical tasks from the preset meta-parameter pool, calculates the similarity between the current task embedding and the historical task embedding, and selects the meta-parameter with the highest similarity as the initialization parameter for the current model training, thereby providing a parameter basis with prior experience for different risk control tasks.

[0220] In the model training phase, the initialization parameters are used to construct task-specific classifiers and meta-classifiers. The task-specific classifiers predict the output of the current customer data, such as credit score level, fraud tendency, etc., and construct task-specific loss in combination with the real label; the meta-classifier models the behavior patterns shared between different customer tasks, such as the correlation between high-frequency transactions and overdue risks, extracts meta-knowledge, and calculates meta-knowledge loss. Combined with the meta-knowledge loss and the task loss, the total loss is generated by adaptive weight fusion, and the parameters of the two types of classifiers are updated through the back propagation mechanism, so that the model not only learns the details of the current task, but also retains the general judgment ability across tasks.

[0221] In the practical deployment phase, the system receives the full-modality data stream of a certain customer, repeatedly executes feature extraction, weighted fusion, prototype construction, etc. processes, inputs the cross-modality prototype into the trained task-specific classifier, and quickly gives the prediction result, such as "high-risk fraud behavior warning" or "credit score: 640". The prediction result is processed by the post-processing module to analyze the threshold and map the business label, and finally returned to the financial risk control middle platform or the automatic approval process, providing the basis for loan approval, fund freezing, or manual review.

[0222] The embodiment can dynamically perceive the importance difference of different modalities in different tasks by joint processing of multi-modal feature extraction and gating network, and avoid the inhibition of fixed weight structure on the prediction performance of the model. The aggregation operation of the intra-modal prototype compresses the complex modal feature distribution into a representative central representation, improves the representation density and cross-modal information alignment efficiency. After introducing the fusion weight coefficient, the multiple modal prototypes are weighted and fused, further enhancing the adaptability of the model when facing modal heterogeneity tasks. Finally, the cross-modal prototypes are discriminated by the updated task-specific classifier, and the application result output is completed through the decoding module, so as to realize the accurate processing of the target task under the condition of heterogeneous multi-modal input.

[0223] In an embodiment, a data processing apparatus based on adaptive meta-learning is provided, which corresponds to the data processing method based on adaptive meta-learning in the above embodiment. Referring to Figure 3 , Figure 3 The functional module schematic diagram of a preferred embodiment of the data processing apparatus based on adaptive meta-learning of the application. Multi-modal feature extraction module 10, modal adaptive weighting module 20, cross-modal prototype fusion module 30, task adaptive parameter retrieval module 40, classifier joint training module 50 and target task reasoning module 60. The detailed description of each functional module is as follows:

[0224] The multi-modal feature extraction module 10 is used to obtain input data of different modalities and extract initial modal features of the input data of each modality respectively;

[0225] The modal adaptive weighting module 20 is used to generate a modal confidence score based on the initial modal features through a gating network, and to perform weighting processing on the initial modal features based on the modal confidence score to obtain weighted modal features;

[0226] The cross-modal prototype fusion module 30 is used to generate intra-modal prototypes corresponding to each modality according to the weighted modal features, and to generate cross-modal prototypes by weighted fusion of multiple intra-modal prototypes based on a fusion weight coefficient;

[0227] The task adaptive parameter retrieval module 40 is used to encode the current learning task to obtain a task embedding vector, and to retrieve a target meta-parameter in a preset meta-parameter pool according to the task embedding vector, and to use the target meta-parameter as an initial value of the model parameter;

[0228] The classifier joint training module 50 is used to establish a task-specific classifier and a meta-classifier using the initial value of the model parameter, and to update the parameters of the task-specific classifier and the meta-classifier based on the cross-modal prototypes;

[0229] The target task reasoning module 60 is configured to process the target task by using the updated task-specific classifier, and generate a target task processing result.

[0230] In an embodiment, the multi-modal feature extraction module 10 is specifically configured to:

[0231] receive original input data containing image modal data, text modal data and audio modal data;

[0232] assign the image modal data to an image processing channel, and process the image modal data by using a convolution feature extractor to obtain image initial modal features;

[0233] assign the text modal data to a text processing channel, and process the text modal data by using a sequence feature extractor to obtain text initial modal features;

[0234] assign the audio modal data to an audio processing channel, and process the audio modal data by using a spectrum feature extractor to obtain audio initial modal features;

[0235] perform normalization processing on the image initial modal features, the text initial modal features and the audio initial modal features;

[0236] unify the normalized image initial modal features, the normalized text initial modal features and the normalized audio initial modal features to the same dimensional space to form the initial modal features.

[0237] In an embodiment, the modal adaptive weighting module 20 is specifically configured to:

[0238] for the initial modal features of each modal, apply a weight matrix corresponding to the modal to perform linear transformation to obtain linear transformation features;

[0239] add a bias term corresponding to the modal to the linear transformation features to obtain bias adjustment features;

[0240] apply a nonlinear activation function to the bias adjustment features to generate activation features;

[0241] apply a normalization function to the activation features to generate the modal confidence score;

[0242] multiply the initial modal features of each modal by the corresponding modal confidence score to obtain weighted features of the corresponding modal;

[0243] combine the weighted features of all the modals to obtain the weighted modal features.

[0244] In an embodiment, the cross-modal prototype fusion module 30 is specifically configured to:

[0245] obtain the weighted modal features of each modal;

[0246] determine the average value of the weighted modal features of each modality, generate an intra-modal prototype of the corresponding modality;

[0247] dynamically generate a fusion weight coefficient of each intra-modal prototype through a meta-learning mechanism;

[0248] multiply the intra-modal prototype of each modality by the corresponding fusion weight coefficient to obtain a weighted intra-modal prototype;

[0249] perform weighted summation on all weighted intra-modal prototypes to obtain the cross-modal prototype.

[0250] In an embodiment, the task adaptive parameter retrieval module 40 is specifically configured to:

[0251] obtain task feature data of a current learning task;

[0252] extract a core feature representation of the task feature data;

[0253] generate a task embedding vector by processing the core feature representation through a task encoder;

[0254] obtain a plurality of historical task embedding vectors in a meta-parameter pool;

[0255] determine the similarity between the task embedding vector and each historical task embedding vector;

[0256] select a meta-parameter corresponding to the highest similarity as a target meta-parameter;

[0257] use the target meta-parameter as an initial value of the model parameter.

[0258] In an embodiment, the classifier joint training module 50 is specifically configured to:

[0259] initialize a task-specific classifier and a meta-classifier using the initial value of the model parameter;

[0260] process the cross-modal prototype through the task-specific classifier to obtain a task-specific prediction result;

[0261] determine a task-specific loss according to the task-specific prediction result and a real label corresponding to the cross-modal prototype;

[0262] analyze an inter-task commonness mode based on the cross-modal prototype through the meta-classifier to obtain a meta-knowledge analysis result;

[0263] determine a meta-knowledge loss according to the meta-knowledge analysis result;

[0264] obtain a meta-knowledge loss weight coefficient, and multiply the meta-knowledge loss by the meta-knowledge loss weight coefficient to obtain a weighted meta-knowledge loss;

[0265] adding the task-specific loss and the weighted meta-knowledge loss to generate a total loss;

[0266] determining a gradient according to the total loss;

[0267] updating parameters of the task-specific classifier and the meta-classifier through back propagation based on the gradient.

[0268] In an embodiment, the target task reasoning module 60 is specifically configured to:

[0269] obtain multi-modal input data of a target task;

[0270] perform feature extraction on the multi-modal input data of the target task to obtain initial modal features of the target task;

[0271] generate target task modal confidence scores through a gating network based on the initial modal features of the target task, and perform weighted processing on the initial modal features of the target task based on the target task modal confidence scores to obtain weighted modal features of the target task;

[0272] determine an average value of the weighted modal features of the target task of each modality to generate an intra-modal prototype of the target task corresponding to the modality;

[0273] perform weighted fusion on the intra-modal prototypes of the target task corresponding to the modalities based on a fusion weight coefficient to generate a cross-modal prototype of the target task;

[0274] input the cross-modal prototype of the target task into the updated task-specific classifier to obtain a target task prediction result;

[0275] perform decoding processing on the target task prediction result to output the target task prediction result as a target task processing result.

[0276] In one embodiment, a computer device is provided, which can be a server. An internal structure diagram of the computer device can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external user terminal through a network connection. The computer program is executed by the processor to implement the functions or steps of the server side of the data processing method based on adaptive meta-learning.

[0277] In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of a data processing method based on adaptive meta-learning on the user terminal side.

[0278] In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0279] Obtain input data of different modalities, and extract initial modality features of the input data of each modality, respectively;

[0280] Based on the initial modality features, generate modality confidence scores through a gating network, and perform weighted processing on the initial modality features based on the modality confidence scores to obtain weighted modality features;

[0281] According to the weighted modality features, generate intra-modality prototypes corresponding to each modality, and weight and fuse multiple intra-modality prototypes based on a fusion weight coefficient to generate cross-modality prototypes;

[0282] Encode a current learning task to obtain a task embedding vector, and in a preset meta-parameter pool, retrieve a target meta-parameter according to the task embedding vector, and use the target meta-parameter as an initial value of a model parameter;

[0283] Establish a task-specific classifier and a meta-classifier using the initial value of the model parameter, and update parameters of the task-specific classifier and the meta-classifier based on the cross-modality prototypes;

[0284] Process a target task through the updated task-specific classifier to generate a target task processing result.

[0285] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0286] Obtain input data of different modalities, and extract initial modality features of the input data of each modality, respectively;

[0287] Based on the initial modal feature, a modal confidence score is generated through a gating network, and the initial modal feature is weighted based on the modal confidence score to obtain a weighted modal feature;

[0288] According to the weighted modal feature, an intra-modal prototype corresponding to each modal is generated, and a plurality of intra-modal prototypes are weighted and fused based on a fusion weight coefficient to generate a cross-modal prototype;

[0289] A current learning task is encoded to obtain a task embedding vector, and a target meta-parameter is retrieved from a preset meta-parameter pool according to the task embedding vector, and the target meta-parameter is used as an initial value of a model parameter;

[0290] A task-specific classifier and a meta-classifier are established using the initial value of the model parameter, and the parameters of the task-specific classifier and the meta-classifier are updated based on the cross-modal prototype;

[0291] A target task processing result is generated by processing the target task through the updated task-specific classifier.

[0292] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the descriptions of the server side and the user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0293] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0294] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0295] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data processing method based on adaptive meta-learning, characterized in that, The method comprises the following steps: obtaining input data of different modalities and extracting initial modality features of the input data of each modality respectively; generating a modality confidence score based on the initial modality features through a gating network, and performing weighted processing on the initial modality features based on the modality confidence score to obtain weighted modality features; generating an intra-modality prototype corresponding to each modality according to the weighted modality features, and generating a cross-modality prototype by weighted fusion of multiple intra-modality prototypes based on a fusion weight coefficient; encoding a current learning task to obtain a task embedding vector, and retrieving a target meta-parameter from a preset meta-parameter pool according to the task embedding vector to take the target meta-parameter as an initial value of a model parameter; establishing a task-specific classifier and a meta-classifier by using the initial value of the model parameter, and updating parameters of the task-specific classifier and the meta-classifier based on the cross-modality prototype; processing a target task through the updated task-specific classifier to generate a target task processing result; the method of generating a modality confidence score based on the initial modality features through a gating network comprises: applying a weight matrix of a corresponding modality to each modality feature to perform linear transformation and obtain linear transformation features; adding a bias term of the corresponding modality to the linear transformation features to obtain bias adjustment features; applying a nonlinear activation function to the bias adjustment features to generate activation features; applying a normalization function to the activation features to generate the modality confidence score; the method of generating an intra-modality prototype corresponding to each modality according to the weighted modality features comprises: obtaining weighted modality features of each modality; determining the average value of the weighted modality features of each modality to generate an intra-modality prototype of the corresponding modality; the method of retrieving a target meta-parameter from a preset meta-parameter pool according to the task embedding vector to take the target meta-parameter as an initial value of a model parameter comprises: obtaining multiple historical task embedding vectors in the meta-parameter pool; determining the similarity between the task embedding vector and each historical task embedding vector; selecting a meta-parameter corresponding to the highest similarity as the target meta-parameter; taking the target meta-parameter as the initial value of the model parameter.

2. The data processing method based on adaptive meta-learning according to claim 1, wherein, The method of obtaining input data of different modalities and extracting initial modality features of the input data of each modality comprises: receiving original input data containing image modality data, text modality data and audio modality data; allocating the image modality data to an image processing channel to obtain image initial modality features through a convolution feature extractor; allocating the text modality data to a text processing channel to obtain text initial modality features through a sequence feature extractor; allocating the audio modality data to an audio processing channel to obtain audio initial modality features through a spectrum feature extractor; performing normalization processing on the image initial modality features, the text initial modality features and the audio initial modality features; unifying the normalized image initial modality features, the normalized text initial modality features and the normalized audio initial modality features to the same dimensional space to form the initial modality features.

3. The data processing method based on adaptive meta-learning according to claim 1, wherein, The initial modality features are weighted based on the modality confidence scores to obtain weighted modality features, including: The initial modality features of each modality are multiplied by the corresponding modality confidence score to obtain the weighted features of the corresponding modality; The weighted features of all modalities are combined to obtain the weighted modality features.

4. The data processing method based on adaptive meta-learning according to claim 1, wherein, Based on the fusion weight coefficient, the intra-modal prototypes of multiple modalities are weighted and fused to generate cross-modal prototypes, including: The fusion weight coefficient of each intra-modal prototype is dynamically generated through the meta-learning mechanism; Each intra-modal prototype of each modality is multiplied by the corresponding fusion weight coefficient to obtain the weighted intra-modal prototype; The weighted sum of all weighted intra-modal prototypes is obtained to obtain the cross-modal prototype.

5. The data processing method based on adaptive meta-learning according to claim 1, wherein, The current learning task is encoded to obtain a task embedding vector, including: Obtain the task feature data of the current learning task; Extract the core feature representation of the task feature data; The task embedding vector is generated by processing the core feature representation through the task encoder.

6. The data processing method based on adaptive meta-learning according to claim 1, wherein, The task-specific classifier and the meta-classifier are established using the initial value of the model parameter, and the parameters of the task-specific classifier and the meta-classifier are updated based on the cross-modal prototype, including: The task-specific classifier and the meta-classifier are initialized using the initial value of the model parameter; The task-specific prediction result is obtained by processing the cross-modal prototype through the task-specific classifier; The task-specific loss is determined according to the task-specific prediction result and the real label corresponding to the cross-modal prototype; The meta-knowledge analysis result is obtained by analyzing the common mode between tasks based on the cross-modal prototype through the meta-classifier; The meta-knowledge loss is determined according to the meta-knowledge analysis result; The meta-knowledge loss weight coefficient is obtained, and the meta-knowledge loss is multiplied by the meta-knowledge loss weight coefficient to obtain the weighted meta-knowledge loss; The task-specific loss and the weighted meta-knowledge loss are added to generate the total loss; The gradient is determined according to the total loss; Based on the gradient, the parameters of the task-specific classifier and the meta-classifier are updated through back propagation.

7. The data processing method based on adaptive meta-learning according to claim 1, wherein, The target task processing result is generated by processing the target task through the updated task-specific classifier, including: Obtain the multi-modal input data of the target task; The target task initial modality features are obtained by performing feature extraction on the multi-modal input data of the target task; Based on the target task initial modality features, the target task modality confidence score is generated through the gating network, and the target task initial modality features are weighted based on the target task modality confidence score to obtain the target task weighted modality features; The average value of the target task weighted modality features of each modality is determined to generate the target task intra-modal prototype of the corresponding modality; Based on the fusion weight coefficient, the target task intra-modal prototypes corresponding to each modality are weighted and fused to generate the target task cross-modal prototype; The target task prediction result is obtained by inputting the target task cross-modal prototype into the updated task-specific classifier; The target task prediction result is decoded and output as the target task processing result.

8. A data processing apparatus based on adaptive meta-learning, characterized by, The data processing apparatus based on adaptive meta-learning includes: The multi-modal feature extraction module is configured to obtain input data of different modalities and extract initial modality features of the input data of each modality, respectively. The modality adaptive weighting module is configured to generate a modality confidence score based on the initial modality features through a gating network and perform weighting processing on the initial modality features based on the modality confidence score to obtain weighted modality features. The cross-modality prototype fusion module is configured to generate intra-modality prototypes corresponding to each modality based on the weighted modality features and fuse the intra-modality prototypes of multiple modalities based on a fusion weight coefficient to generate a cross-modality prototype. The task adaptive parameter retrieval module is configured to encode a current learning task to obtain a task embedding vector and retrieve a target meta-parameter from a preset meta-parameter pool based on the task embedding vector, and use the target meta-parameter as an initial value of a model parameter. The classifier joint training module is configured to establish a task-specific classifier and a meta-classifier based on the initial value of the model parameter and update parameters of the task-specific classifier and the meta-classifier based on the cross-modality prototype. The target task reasoning module is configured to process a target task through the updated task-specific classifier to generate a target task processing result. The modality adaptive weighting module is specifically configured to: apply a weight matrix of a corresponding modality to each initial modality feature to perform linear transformation and obtain linearly transformed features; add a bias term of the corresponding modality to the linearly transformed features to obtain bias-adjusted features; apply a nonlinear activation function to the bias-adjusted features to generate activated features; and apply a normalization function to the activated features to generate the modality confidence score. The cross-modality prototype fusion module is specifically configured to: obtain weighted modality features of each modality; determine an average value of the weighted modality features of each modality to generate an intra-modality prototype of the corresponding modality; and The task adaptive parameter retrieval module is specifically configured to: obtain multiple historical task embedding vectors in the meta-parameter pool; determine a similarity between the task embedding vector and each historical task embedding vector; select a meta-parameter corresponding to the highest similarity as a target meta-parameter; and use the target meta-parameter as an initial value of a model parameter.

9. A computer device, comprising: The computer device includes a memory, a processor, and a data processing program based on adaptive meta-learning stored on the memory and running on the processor. When the data processing program based on adaptive meta-learning is executed by the processor, the steps of the data processing method based on adaptive meta-learning in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The storage medium stores a data processing program based on adaptive meta-learning. When the data processing program based on adaptive meta-learning is executed by the processor, the steps of the data processing method based on adaptive meta-learning in any one of claims 1-7 are implemented.

Images