Multi-modal general-special model collaborative reasoning method based on dynamic routing mechanism

Abstract

The embodiment of the application provides a multi-modal general-purpose model collaborative reasoning method based on a dynamic routing mechanism. The method receives a task instruction and multi-modal data corresponding to the task instruction, determines a task type corresponding to the task instruction, and extracts data features of the multi-modal data. A general-purpose model is used to fuse the data features to generate general fusion features. According to a preset processing mapping relationship, a task-specific model, a modal processing procedure, and fusion feature weights of the current task type are matched. The data features and the general fusion features are transmitted to the task-specific model through a routing node. The task-specific model processes the data features and the general fusion features according to the modal processing procedure and the fusion feature weights to obtain specific fusion features. The specific fusion features are processed by a task execution component to obtain a task result corresponding to the task instruction. The method can effectively solve the lack of model flexibility in multi-modal data processing and significantly improve the convenience of semantic fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal general-purpose model collaborative inference method based on a dynamic routing mechanism. BACKGROUND

[0002] In the process of processing multi-modal data, a cross-modal feature fusion mode can be performed through a static model architecture. Specifically, different modal data can be processed through a network structure with fixed parameters. However, in the cross-modal feature fusion mode through the static model architecture, resource idling or overload phenomenon is likely to occur when processing heterogeneous multi-modal data, affecting the overall efficiency.

[0003] In the prior art, different modal data can be independently processed by constructing task-specific models, the feature weights can be dynamically adjusted through an attention mechanism, and a resource monitoring module can be introduced to allocate computing resources. However, in the mode of independently processing different modal data by constructing task-specific models, the lack of collaboration between different models leads to low information utilization; in the mode of dynamically adjusting feature weights through an attention mechanism, the optimization dimension is relatively single; in the mode of introducing a resource monitoring module to allocate computing resources, there is a lack of dynamic routing strategy, and the resource granularity is relatively rough, resulting in difficulty in coordinating the three-dimensional dynamics between task requirements, modal features, and resource allocation, and the processing parameters cannot be optimized in real time during the model inference process, which reduces the efficiency of processing multi-modal data. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a multi-modal general-purpose model collaborative inference method based on a dynamic routing mechanism, which can effectively solve the problems of semantic fusion difficulty, insufficient model flexibility, and failure to maximize the use of computing resources in the process of processing multi-modal data in traditional technologies, and significantly improve the convenience of semantic fusion and the utilization rate of computing resources.

[0005] To solve at least one of the above problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a multi-modal general-purpose model collaborative inference method based on a dynamic routing mechanism, comprising:

[0007] receiving a task instruction and multi-modal data corresponding to the task instruction, determining a task type corresponding to the task instruction, extracting data features corresponding to the multi-modal data, and fusing the data features based on a task general-purpose model to obtain general-purpose fusion features, the task type including a classification task, a generation task, and a retrieval task;

[0008] receiving a preset processing mapping relationship, determining a task-specific model, a modal processing flow, and a fusion feature weight corresponding to the current task instruction based on the task type in the preset processing mapping relationship;

[0009] The data features and the general fusion features are sent to the task-specific model through the routing node, so that the task-specific model processes the data features and the general fusion features based on the modal processing procedure and the fusion feature weight, obtains specific fusion features, and processes the specific fusion features through a task execution component to obtain a task result corresponding to the task instruction.

[0010] Further, the data features are respectively mapped to a preset semantic space in the task general model to obtain mapped data features, the similarity between the mapped data features is determined, the mapped data features are assigned a general feature weight based on the similarity, and the similarity is positively correlated with the feature weight.

[0011] Based on the feature weight, the data features are fused through the task general model to obtain general fusion features.

[0012] Further, before receiving the preset processing mapping relationship, the method further comprises:

[0013] A preset processing mapping relationship library is constructed, and the preset processing mapping relationship library comprises a combination of the task-specific model, the modal processing procedure and the fusion feature weight corresponding to each task category;

[0014] In the preset processing mapping relationship, the task-specific model, the modal processing procedure and the fusion feature weight corresponding to the current task instruction are determined based on the task type, comprising:

[0015] Based on the task type, the task-specific model, the modal processing procedure and the fusion feature weight matched with the current task type are determined from the preset processing mapping relationship library.

[0016] Further, when the task type corresponding to the task instruction is a classification task, the specific fusion features are processed through a classification algorithm to obtain a classification label corresponding to the task instruction.

[0017] When the task type corresponding to the task instruction is a generation task, the specific fusion features are processed through a generation model to generate at least one of a text generation result, an image generation result and an audio generation result corresponding to the task instruction.

[0018] When the task type corresponding to the task instruction is a retrieval task, the specific fusion features are processed through a retrieval algorithm to obtain a retrieval result corresponding to the task instruction.

[0019] Further, after determining the task-specific model, the modal processing procedure and the fusion feature weight corresponding to the current task instruction based on the task type in the preset processing mapping relationship, the method further comprises:

[0020] determine a complexity level of the task instruction, wherein the complexity level is determined based on a data scale of the multi-modal data and a task type;

[0021] determine a computing resource required for computing the multi-modal data in a process of completing the task instruction based on the multi-modal data, determine a resource allocation strategy corresponding to the current task instruction based on the complexity level and the computing resource, wherein the computing resource includes a memory resource, a display memory resource, and a central processing unit computing capability requirement;

[0022] send the data features and the general fusion features to the task-specific model through the routing node, including:

[0023] send the data features and the general fusion features to the corresponding task-specific model through the routing node based on the resource allocation strategy.

[0024] Further, after sending the data features and the general fusion features to the task-specific model through the routing node, further including:

[0025] monitor the execution performance indicators of each task-specific model in real time, the execution performance indicators including accuracy, recall rate, execution time, and consumption of computing resources;

[0026] based on the execution performance indicators, dynamically adjust the processing parameters in the modality processing flow in the preset processing mapping relationship.

[0027] Further, further including: monitoring the computing resource consumption of each task-specific model in the process of processing the data features and the general fusion features in real time, and monitoring the task progress state corresponding to the task instruction in real time;

[0028] when it is monitored that there is a task-specific model that has completed processing of the data features and the general fusion features and there is remaining computing resource, and other task-specific models have not completed processing of the data features and the general fusion features, allocate the remaining computing resource to the other task-specific models.

[0029] In a second aspect, the application provides a multi-modal task-specific model collaborative reasoning device based on a dynamic routing mechanism, including:

[0030] a receiving module configured to receive a task instruction and multi-modal data corresponding to the task instruction, determine a task type corresponding to the task instruction, extract data features corresponding to the multi-modal data, and fuse the data features based on a task general model to obtain general fusion features, the task type including a classification task, a generation task, and a retrieval task;

[0031] a mapping module configured to receive a preset processing mapping relationship, determine a task-specific model corresponding to the current task instruction, a modality processing flow, and a fusion feature weight in the preset processing mapping relationship based on the task type;

[0032] The processing module is configured to send the data features and the general fusion features to the task-specific model through the routing node, so that the task-specific model processes the data features and the general fusion features based on the modality processing procedure and the fusion feature weight, obtains the specific fusion features, and processes the specific fusion features through the task execution component to obtain the task result corresponding to the task instruction.

[0033] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for collaborative inference of a multi-modal general-special model based on a dynamic routing mechanism when executing the program.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the method for collaborative inference of a multi-modal general-special model based on a dynamic routing mechanism.

[0035] In a fifth aspect, the present application provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions are executable on a processor to implement the steps of the method for collaborative inference of a multi-modal general-special model based on a dynamic routing mechanism.

[0036] According to the above technical solution, the present application provides a method for collaborative inference of a multi-modal general-special model based on a dynamic routing mechanism. The method receives a task instruction and multi-modal data corresponding to the task instruction, determines a task type corresponding to the task instruction, and extracts data features of the multi-modal data. The method generates general fusion features by fusing the data features through a task general model, matches a task-specific model, a modality processing procedure, and a fusion feature weight of the current task type according to a preset processing mapping relationship, transmits the data features and the general fusion features to the task-specific model through a routing node, and makes the task-specific model process the data features and the general fusion features according to the modality processing procedure and the fusion feature weight to obtain specific fusion features. The method processes the specific fusion features through a task execution component to obtain a task result corresponding to the task instruction. The method can dynamically determine a task-specific model and a fusion feature weight according to a task instruction, can process multiple tasks simultaneously according to the same set of multi-modal data, improves the inference efficiency and accuracy of different task classifications, allocates computing resources on demand through a routing node, reduces redundant processing, and improves the efficiency and overall performance of multi-modal semantic fusion. The method can effectively solve the problems of traditional technologies, such as difficulty in semantic fusion when processing multi-modal data, insufficient model flexibility, and failure to maximize the use of computing resources, significantly improves the convenience of semantic fusion, and improves the utilization rate of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A flowchart of a multi-modal general-special model collaborative reasoning method based on a dynamic routing mechanism in an embodiment of the present application;

[0039] Figure 2 A structural diagram of a multi-modal general-special model collaborative reasoning device based on a dynamic routing mechanism in an embodiment of the present application;

[0040] Figure 3 A structural diagram of an electronic device in an embodiment of the present application.

[0041] Reference signs:

[0042] Electronic device 9600, central processor 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] The acquisition, storage, use, processing, etc. of data in the technical solutions of the present application all comply with the relevant provisions of national laws and regulations.

[0045] In the prior art, multi-modal data of different modalities have their own semantic representation methods, and it is difficult to effectively fuse and collaboratively reason them. In the face of different task requirements and data analysis changes, the existing multi-modal model is difficult to quickly adjust its structure and parameters, resulting in unstable performance in different scenarios. Due to the lack of effective task allocation mechanism, the multi-modal model may waste resources on some unnecessary calculations, affecting the reasoning efficiency.

[0046] In order to effectively solve the problems of traditional technology in multi-modal data processing, such as semantic fusion difficulty, insufficient model flexibility, and failure to maximize the use of computing resources, and significantly improve the convenience of semantic fusion and the utilization rate of computing resources, an embodiment of a multi-modal general-special model collaborative reasoning method based on a dynamic routing mechanism is provided, as shown in Figure 1 The multi-modal general-special model collaborative reasoning method based on the dynamic routing mechanism specifically includes the following contents:

[0047] Step S101: receiving a task instruction and multi-modal data corresponding to the task instruction, determining a task type corresponding to the task instruction, extracting data features corresponding to the multi-modal data, and fusing the data features based on a general task model to obtain general fusion features.

[0048] The task type includes a classification task, a generation task, and a retrieval task.

[0049] Optionally, the embodiment receives a task instruction and multi-modal data corresponding to the task instruction, wherein the multi-modal data includes data of multiple modal types, including but not limited to at least two of image data, text data, and audio data.

[0050] At the same time, the task type corresponding to the task instruction is determined according to the task instruction, wherein the task type includes but is not limited to a classification task, a generation task, and a retrieval task.

[0051] In addition, the multi-modal data is subjected to data feature extraction to obtain data features, wherein the data features include at least two of text features, image features, and audio features.

[0052] Different feature extraction methods can be used according to the different data types of the multi-modal data. The image features of the image data can be extracted through a convolutional neural network, wherein the convolutional layers and the pooling layers of the convolutional neural network can learn the edge, texture, shape, and other features in the image data to generate image features.

[0053] The text data is extracted through a recurrent neural network or a transformer model to obtain text features, wherein the recurrent neural network can capture sequence information in the text data, and the text features are obtained according to the sequence information. The transformer model can process the text in parallel through a self-attention mechanism to generate a semantic feature vector (i.e., text features).

[0054] The audio data is extracted through an audio feature extraction algorithm to obtain audio features.

[0055] Further, the data features are fused by a task general model to obtain general fusion features, wherein the task general model is a model with strong generalization ability and can learn common features between different modal data, and the way of obtaining general fusion features includes but is not limited to splicing fusion and dimension reduction fusion, wherein the splicing fusion is to directly splice the image features, text features and audio feature vectors required for fusion into a long vector to form general fusion features, and the dimension reduction splicing is to splice after dimension reduction processing of each feature data to obtain general fusion features.

[0056] Further, the task general model can also map data features of different modalities to a unified semantic space, calculate semantic correlation weights between feature vectors to reflect the correlation degree between multi-modal data, and fuse data features according to weighted summation or attention mechanism-based fusion to obtain general fusion features.

[0057] The embodiment realizes the accuracy and consistency of enhancing semantic fusion of multi-modal data, improves the adaptability and flexibility of the task general model to different task types, and improves the efficiency and resource utilization.

[0058] Step S102: receiving a preset processing mapping relationship, determining a task-specific model, a modal processing procedure and a fusion feature weight corresponding to the current task instruction based on the task type in the preset processing mapping relationship.

[0059] Optionally, the embodiment receives a preset processing mapping relationship, wherein the preset processing mapping relationship can be obtained according to a large amount of historical task data and expert experience, and through the preset processing mapping relationship, a suitable task-specific model, modal processing procedure and fusion feature weight can be quickly found for the task instruction.

[0060] The preset processing mapping relationship can be constructed in the following way: a large amount of historical task data of different task types (classification tasks, generation tasks and retrieval tasks) are collected, and the historical task data includes but is not limited to task instructions, multi-modal data samples and corresponding model configurations.

[0061] The historical task data are analyzed and mined, that is, the association mode between the task type and the optimal model configuration can be found from the historical task data, the mapping rule is constructed according to the association mode, the task type is taken as the input, and the optimal task-specific model, modal processing procedure and fusion feature weight are taken as the output, and the preset processing mapping relationship is formed.

[0062] The embodiment realizes the dynamic adaptability to different task types, improves the efficiency and accuracy of multi-modal collaborative reasoning, and improves the optimized utilization of resources and the overall performance of task processing.

[0063] Step S103: Send the data features and general fusion features to the task-specific model through the routing node, so that the task-specific model processes the data features and general fusion features based on the modal processing flow and fusion feature weights to obtain the special fusion features, and processes the special fusion features through the task execution component to obtain the task result corresponding to the task instruction.

[0064] Optionally, in this embodiment, the extracted data features and general fusion features are sent to the task-specific model through a routing node. The routing node has functions including, but not limited to, data transmission and route selection. Data transmission is used to ensure that the data features and general fusion features are accurately transmitted from the feature extraction end to the task-specific model. Route selection is used to dynamically select the optimal transmission path based on the current task instructions and multimodal data, thereby improving the efficiency and reliability of data transmission.

[0065] In addition, after receiving the data, the task-specific model further processes the data features and general fusion features according to the received modality processing flow and fusion feature weights to obtain the task-specific fusion features. The modality processing flow may include, but is not limited to, feature preprocessing and feature weighting steps.

[0066] Feature preprocessing is used to normalize, fill in missing values, and handle outliers for each multimodal data feature to ensure the quality and consistency of the data features. Feature weighted fusion is used to fuse data features and general fusion features by weighted summation according to the weight of the fusion features to generate special fusion features.

[0067] In addition, the obtained special fusion features are input into the task execution component for processing to obtain the task result corresponding to the task instruction. The task execution component can obtain the task result through different processing methods according to different task types.

[0068] This embodiment enhances the adaptability of multimodal data to dynamic changes according to task instructions, improves the accuracy of collaborative reasoning and the robustness of semantic fusion, and enhances the utilization efficiency of computing resources and the execution efficiency of the overall reasoning process.

[0069] This embodiment enables efficient processing of multimodal data, determines the task-specific model corresponding to the task type, the modality processing flow and the fusion feature weights, and outputs task results through routing node transmission, task-specific model processing and task execution component collaboration, thereby meeting the multi-domain multimodal data processing needs and improving performance.

[0070] In some embodiments, general fusion features are obtained based on the fusion of data features according to the general task model, including:

[0071] The data features are respectively mapped to a preset semantic space in the task general model to obtain mapped data features, the similarity between the mapped data features is determined, the general feature weight is assigned to the mapped data features based on the similarity, and the similarity is positively correlated with the feature weight;

[0072] Based on the feature weight, the data features are fused through the task general model to obtain general fusion features.

[0073] Optionally, the embodiment can map the extracted data features corresponding to the multi-modal data to a preset semantic space in the task general model to obtain mapped data features. The preset semantic space is a high-dimensional and unified semantic representation space, which is used to eliminate the semantic difference between different modal data, so that the features from different modalities can be compared and fused in a common space.

[0074] The mapping method can be different for different modal data, including:

[0075] Image feature mapping: image features are mapped into a preset semantic space through a pre-trained visual embedding model (such as visual Transformer), wherein the visual embedding model can be pre-trained on a large-scale image dataset to learn the mapping relationship between image features and semantic concepts, i.e., mapped image features.

[0076] Text feature mapping: text features of text data can be mapped into a preset semantic space through a word vector model (such as Word2Vec, GloVe) and semantic role labeling, wherein the word vector model can capture the semantic similarity and context relationship between words, and the semantic role labeling can extract the core semantic components in the sentence, i.e., the mapped text features.

[0077] Audio feature mapping: audio features can be mapped into a semantic space through audio semantic analysis technology (such as an audio embedding model), wherein the audio embedding model converts the spectrum, timing and other features of the audio into a semantic-related representation, i.e., the mapped audio features.

[0078] In addition, in the preset semantic space, the similarity between the mapped data features is calculated to measure the correlation of multi-modal data of different modalities at the semantic level. The similarity calculation method includes but is not limited to cosine similarity, Euclidean distance, etc.

[0079] In addition, according to the calculated similarity, a general feature weight is assigned to the mapped data features, wherein the similarity is positively correlated with the feature weight, i.e., the higher the similarity of a feature, the greater the feature weight, and the assignment method of the feature weight can adopt linear normalization processing to map the similarity value to the weight interval.

[0080] Further, according to the assigned feature weight, the data features are fused by the task general model to obtain general fusion features, wherein the fusion manner includes but is not limited to weighted summation and feature splicing.

[0081] Further, the distribution difference of different modal data in the preset semantic space can be reduced in an adversarial training manner to enhance the consistency and accuracy of feature mapping.

[0082] The embodiment realizes mapping of multi-modal data features to a unified preset semantic space, and assigns and fuses feature weights based on similarity, which can effectively integrate semantic information of different modal data, improve the performance of various task types such as classification tasks, generation tasks and retrieval tasks, and at the same time, the optimized calculation process and resource utilization manner make the process of processing various task types more efficient and economical.

[0083] In some embodiments, before receiving the preset processing mapping relationship, further comprising:

[0084] A preset processing mapping relationship library is constructed, and the preset processing mapping relationship library includes combinations of task-specific models, modal processing procedures and fusion feature weights corresponding to each task category;

[0085] In the preset processing mapping relationship, the task-specific model, the modal processing procedure and the fusion feature weight corresponding to the current task instruction are determined based on the task type, comprising:

[0086] Based on the task type, the task-specific model, the modal processing procedure and the fusion feature weight matched with the current task type are determined from the preset processing mapping relationship library.

[0087] Optionally, before receiving the preset processing mapping relationship, the embodiment constructs a preset processing mapping relationship library, wherein the preset processing mapping relationship library collects combinations of task-specific models, modal processing procedures and fusion feature weights corresponding to each task category.

[0088] A large amount of multi-modal data samples in different fields and corresponding annotation information are collected, including examples of various task types such as classification tasks, generation tasks and retrieval tasks. At the same time, relevant task execution logs and model performance evaluation data are collected to understand the performance of different task-specific models in various task types.

[0089] The collected multi-modal data samples and labeled information are classified, summarized into different task types, and the features of each multi-modal data sample are extracted, including but not limited to the target of the task, the input modal type, the output requirement, the semantic analysis of the description of each task through natural language processing technology, and the core semantic features of the description of the task. At the same time, the input and output examples of the task are analyzed to extract data modal features and task structure features.

[0090] For each task type, a plurality of candidate task-specific models are trained, which can be based on different algorithms and architectures, such as deep learning models and traditional machine learning models. During the training process, the performance of different task-specific models can be evaluated through cross-validation, including but not limited to accuracy, recall, F1 value, etc. At the same time, the performance of each task-specific model under different modal processing processes and fusion feature weight combinations is recorded.

[0091] According to the features of the task samples and the evaluation results of the models, a mapping relationship between the task type and the task-specific model, the modal processing process and the fusion feature weight is established. The mapping relationship can be learned by machine learning methods such as decision trees, support vector machines, neural networks, etc. The mapping relationship between the features of the task samples and the best task-specific model configuration is learned, and the established mapping relationship is stored in a preset processing mapping relationship library.

[0092] In addition, in the preset processing mapping relationship library, based on the determined task type, a plurality of task-specific models, modal processing processes and fusion feature weight combinations corresponding to the current task type are found.

[0093] In addition, according to the specific circumstances of the current task instruction, such as the characteristics of the input modal data and the real-time requirements, the best task-specific model, modal processing process and fusion feature weight are selected from the plurality of combinations to determine the task-specific model, modal processing process and fusion feature weight for processing the current task instruction.

[0094] Further, each combination can be evaluated by weighted scoring, and the combination with the highest comprehensive score is selected. The distribution of the fusion feature weight is determined according to the influence degree of each multi-modal data on the execution effect of the task instruction.

[0095] The embodiment realizes efficient and accurate processing of task instructions through the construction and preset processing mapping relationship library, and improves the task execution efficiency and accuracy.

[0096] In some embodiments, the task execution component processes the special fusion features to obtain the task result corresponding to the task instruction, including:

[0097] When the task type corresponding to the task instruction is a classification task, the special fusion features are processed by a classification algorithm to obtain a classification label corresponding to the task instruction.

[0098] When the task type corresponding to the task instruction is a generation task, the special fusion features are processed by a generation model to generate at least one of a text generation result, an image generation result, and an audio generation result corresponding to the task instruction.

[0099] When the task type corresponding to the task instruction is a retrieval task, the special fusion features are processed by a retrieval algorithm to obtain a retrieval result corresponding to the task instruction.

[0100] Optionally, when the task type corresponding to the task instruction is a classification task, the most suitable classification algorithm is selected from a preset processing mapping relationship library according to the task instruction, such as a support vector machine (SVM), a decision tree, a random forest, and a deep neural network classifier.

[0101] For example, in an image classification task, if the special fusion features have a high-dimensional and complex distribution, a deep neural network classifier can be used for classification to obtain a corresponding classification label.

[0102] Further, for a multi-label classification task, a multi-label classification algorithm can be used for processing to obtain a corresponding classification label, wherein the multi-label classification algorithm can simultaneously predict the output of multiple labels. For example, in news classification, a piece of news can involve multiple topics (such as politics, economy, technology, etc.), and the multi-label classification algorithm can comprehensively consider the multi-modal information in the special fusion features to assign multiple suitable topic labels to each piece of news.

[0103] Further, after generating the classification result, a post-processing technique (such as label smoothing and threshold adjustment) is used to optimize the result, reduce the influence of overfitting and noise, and at the same time, a verification mechanism such as cross-validation and contrast testing can be introduced to ensure the accuracy and reliability of the result.

[0104] In addition, for a text generation task, a sequence-to-sequence (Seq2Seq) model, a transformer model, a generative adversarial network (GAN), or the like can be used to obtain a text generation result.

[0105] Taking product description generation as an example, the special fusion features are input into the generation model, so that the generation model automatically generates descriptive text based on the text generation mode and semantic logic learned by the training data. During the generation process, the attention mechanism can be used to focus on key feature information to improve the relevance and coherence of the generated text.

[0106] In the image generation task, the special fusion feature can be processed by a generative adversarial network (GAN), a variational autoencoder (VAE), or the like to obtain an image generation result.

[0107] For example, in the process of obtaining a corresponding image generation result according to a text description, the generation model converts the text semantic information in the special fusion feature into image features, gradually constructs image content consistent with the description, and can also use an adversarial training mechanism to continuously optimize the image generation result in quality and details, making it more realistic and consistent with expectations.

[0108] In addition, for the audio generation task, the special fusion feature can be processed by a WaveNet, Tacotron, or the like to obtain an audio generation result.

[0109] For example, in the process of generating natural and fluent voice audio according to text content, the generation model combines text semantics and voice features in the special fusion feature to generate corresponding audio waveforms, which can focus on details such as voice tone, speed, emotion, and the like to improve the naturalness and expressiveness of the audio generation result.

[0110] In addition, when the task instruction corresponds to a retrieval task, a suitable retrieval algorithm can be selected according to the task instruction, such as a retrieval method based on Euclidean distance, cosine similarity, Hamming distance, or the like.

[0111] For example, in the retrieval of cross-modal image data-text data, the similarity between the special fusion feature and each data item in the database is determined, the retrieval results are sorted according to the similarity, and the data item most matching the query intent is returned.

[0112] In the multi-modal retrieval task, the data features of different modalities are fused and processed to improve the accuracy and comprehensiveness of retrieval through multi-modal information. For example, in the video retrieval task, the visual features, audio features, and text description features of the video are combined for comprehensive retrieval, which can more accurately locate the video segment that meets the query requirements.

[0113] Further, the preliminary retrieval results can be post-processed and optimized, such as result reordering, deduplication, result fusion, and the like, and at the same time, the user feedback mechanism can be combined to improve the relevance and satisfaction of the retrieval results, i.e., according to user behavior data such as clicks, dwell time, and the like, dynamically adjusting the parameters and model configurations of the retrieval algorithm.

[0114] The embodiment realizes efficient processing of the special fusion feature by the task execution component, improves the accuracy and quality of the task result, and enhances the intelligence and adaptability.

[0115] In some embodiments, after determining the task-specific model, the modal processing flow and the fusion feature weight corresponding to the current task instruction based on the task type in the preset processing mapping relationship, the method further comprises:

[0116] determining the complexity level of the task instruction, wherein the complexity level is determined based on the data scale of the multi-modal data and the task type;

[0117] determining the computing resources required for computing the multi-modal data in the process of completing the task instruction based on the multi-modal data, and determining the resource allocation strategy corresponding to the current task instruction based on the complexity level and the computing resources, wherein the computing resources include memory resources, video memory resources and central processing unit computing capability requirements;

[0118] sending the data features and the general fusion features to the task-specific model through the routing node, comprising:

[0119] sending the data features and the general fusion features to the corresponding task-specific model through the routing node based on the resource allocation strategy.

[0120] Optionally, after determining the task-specific model, the modal processing flow and the fusion feature weight in the preset processing mapping relationship, the embodiment needs to further determine the complexity level of the task instruction, wherein the determination of the complexity level comprehensively considers two factors of the data scale of the multi-modal data and the task type.

[0121] The data scale includes the evaluation of at least two multi-modal data among image data, text data and audio data, which needs to be consistent with the input multi-modal data. Among them, the image data can be evaluated by the number, resolution, color depth, etc. of the image data, for example, high-resolution image data has larger data scale and higher processing difficulty than low-resolution image data.

[0122] The length, vocabulary richness and sentence complexity of the text data can be considered, wherein the text data with long text or containing a large number of professional terms and complex sentences has larger data scale and is more complex to process.

[0123] The audio data can analyze the length, sampling rate and channel number of the audio, wherein the audio data with longer length, higher sampling rate and more channels has larger data scale and higher processing difficulty.

[0124] The task type includes the complexity level evaluation of classification tasks, generation tasks and retrieval tasks, wherein the simple classification task in the classification task may only involve a few categories and single modal data, and the complex classification task may involve multiple categories, multi-modal data and class imbalance problems.

[0125] Simple generation tasks in the generation task may only generate short texts or simple graphics, and complex generation tasks may need to generate long, high-quality, and highly relevant texts, images, or audios.

[0126] Simple retrieval tasks in the retrieval task may perform single-modal retrieval in small-scale data sets, and complex retrieval tasks may perform cross-modal retrieval in large-scale data sets, with higher requirements for retrieval accuracy and recall rate.

[0127] Based on the above factors, a complexity evaluation model can be established to quantitatively evaluate the complexity level of the task instruction, and the complexity level can be divided into three levels: low complexity level, medium complexity level, and high complexity level.

[0128] For example, a task instruction containing a small number of low-resolution images and simple classification tasks can be evaluated as a low complexity level, and a task instruction containing a large number of high-resolution images, long text descriptions, and complex generation tasks can be evaluated as a high complexity level.

[0129] In addition, the computing resources required for computing multi-modal data in the process of determining the multi-modal data to complete the task instruction are determined, including but not limited to memory resources, video memory resources, and central processing unit (CPU) computing power requirements.

[0130] According to the task type and data size, the demand for computing resources of each modal data processing link is evaluated, wherein image data processing requires more video memory resources for storing and processing high-resolution image feature maps, and also has certain requirements for GPU computing power; text data processing mainly consumes memory resources and CPU computing power for storing text sequences and performing natural language processing related calculations; audio data processing has certain requirements for memory resources and CPU computing power, especially for long-time audio processing and analysis.

[0131] In addition, based on the complexity level and computing resource requirements, the resource allocation strategy corresponding to the current task instruction is determined, and for a high complexity level task instruction, more computing resources can be allocated to meet its processing requirements, and for a low complexity level task instruction, relatively less resources can be allocated to achieve rational use of resources.

[0132] For example, for a high complexity image generation task, more video memory resources and GPU computing power are allocated to the image generation model to ensure the quality and speed of the generated image, and for a low complexity text classification task, appropriate memory resources and CPU computing power are allocated to the text classification model to avoid waste of resources.

[0133] In addition, the data features and general fusion features are sent to the task-specific model through the routing node, including:

[0134] The routing node determines the transmission priority and transmission path of the data features and the general fusion features according to the resource allocation strategy. For a modal data processing link with high computing resource demand, the routing node preferentially ensures that the required data can be transmitted to the task-specific model processing the data features in a timely and efficient manner.

[0135] According to the complexity level of the task instruction and the resource allocation strategy, the bandwidth and rate of data transmission are dynamically adjusted. For a high-complexity task instruction, more transmission bandwidth is allocated to speed up data transmission and reduce transmission delay, so that the task-specific model can quickly receive data and start processing.

[0136] Further, during data transmission, the routing node is also responsible for managing and scheduling data to avoid data congestion and loss.

[0137] For example, when multiple task instructions transmit data at the same time, the routing node allocates transmission resources reasonably according to the resource allocation strategy and priority of each sub-task, ensuring that each multi-modal data can reach the corresponding task-specific model of the sub-task smoothly.

[0138] Further, a resource demand prediction algorithm is developed to predict the change of computing resource demand of the task at different stages in advance, and according to the prediction result, the resource allocation strategy is adjusted in advance to dynamically allocate or release computing resources for the task, improving the utilization efficiency of resources and the execution efficiency of the task. For example, for a long-running video processing task, it is predicted that the demand for CPU and video memory resources is high during video encoding, and sufficient resources are allocated in advance to ensure that the corresponding task can proceed smoothly.

[0139] The embodiment realizes the determination of the complexity level of the task instruction after determining the task-specific model, the modal processing flow and the fusion feature weight based on the task type in the preset processing mapping relationship, and determines the resource allocation strategy accordingly, and then sends the data features and the general fusion features to the task-specific model through the routing node according to the strategy, realizes the fine resource management and efficient execution of processing, and improves the resource utilization efficiency and overall performance.

[0140] In some embodiments, after sending the data features and the general fusion features to the task-specific model through the routing node, the following steps are further included:

[0141] The execution performance indicators of each task-specific model are monitored in real time, including accuracy, recall rate, execution time and consumption of computing resources.

[0142] Based on the execution performance indicators, the processing parameters in the modal processing flow in the preset processing mapping relationship are dynamically adjusted.

[0143] Optionally, the embodiment monitors the execution performance indicators of each task-specific model in real time after sending the data features and the general fusion features to the task-specific model through the routing node, wherein the execution performance indicators include the accuracy, the recall rate, the execution time and the consumption of computing resources.

[0144] The accuracy is used to measure the matching degree of the task result obtained by the task-specific model and the true result, for example, in a classification task, the accuracy represents the proportion of the number of correctly classified samples to the total number of samples, and in a retrieval task, the accuracy represents the proportion of the number of retrieved relevant results to the total number of retrieved results.

[0145] The recall rate is used to emphasize the coverage degree of the task result obtained by the task-specific model to all true positive examples, for example, in a classification task, the recall rate represents the proportion of the number of correctly classified positive samples to the total number of positive samples, and in a retrieval task, the recall rate represents the proportion of the number of retrieved relevant results to the total number of relevant results.

[0146] The execution time is used to record the time spent by the task-specific model from receiving the data features and the general fusion features to obtaining the task result, reflecting the running efficiency of the task-specific model.

[0147] The consumption of computing resources includes, but is not limited to, memory occupation, video memory occupation, usage rate of central processing unit (CPU) and graphics processing unit (GPU), etc., reflecting the resource utilization of the task-specific model in the process of executing the task instruction.

[0148] Further, the above performance indicator data can be monitored in real time through a special monitoring module, a performance monitoring tool and a log record, etc., and stored in a performance monitoring database for subsequent analysis and processing.

[0149] In addition, a mathematical model or a machine learning algorithm can be established to analyze the relationship between the accuracy, the recall rate, the execution time and the consumption of computing resources and the processing parameters in the modal processing flow. For example, the influence of the word vector dimension in text feature extraction, the convolution kernel size and step size in image feature extraction and other parameters on the accuracy and execution time of the classification task is analyzed.

[0150] According to the requirements of the task instruction and the current resource status, the target of performance optimization is determined, if the task instruction has a higher real-time requirement, the target is to reduce the execution time, if the task instruction pays more attention to the accuracy of the result, the optimization target is to improve the accuracy or the recall rate.

[0151] Meanwhile, a corresponding adjustment strategy can be formulated, such as an optimization algorithm based on gradient descent, genetic algorithm, etc., to determine the adjustment direction and amplitude of the processing parameters.

[0152] According to the adjustment strategy, the processing parameters in the modal processing flow are modified in real time during the execution of the task-specific model. For example, in a classification task, if the current convolution kernel size causes the task-specific model to take too long to execute, but the accuracy improvement is limited, the size of the convolution kernel can be appropriately reduced to improve the running efficiency of the task-specific model.

[0153] The embodiment realizes performance optimization and adaptive adjustment of multi-modal general-special model collaborative reasoning by monitoring the execution performance indicators of each task-specific model in real time and dynamically adjusting the processing parameters of the modal processing flow in the preset processing mapping relationship, so that the task general model and the task-specific model can maintain an efficient running state under different task instructions and different multi-modal data conditions, improve the accuracy, recall rate and execution efficiency, and optimize the utilization of computing resources.

[0154] In some embodiments, based on the execution performance indicators, the processing parameters in the modal processing flow in the preset processing mapping relationship are dynamically adjusted, including:

[0155] The computing resource consumption of each task-specific model in processing the data features and the general fusion features is monitored in real time, and the task progress state corresponding to the task instruction is monitored in real time.

[0156] When it is monitored that there is a task-specific model that has completed processing of the data features and the general fusion features and there is remaining computing resource, and other task-specific models have not completed processing of the data features and the general fusion features, the remaining computing resource is allocated to the other task-specific models.

[0157] Optionally, the embodiment monitors the memory and video memory size occupied by the task-specific model in the processing process in real time. For example, when processing image data, the video memory occupation is monitored to ensure that the video memory does not overflow.

[0158] During monitoring by the performance monitoring tool, the CPU and GPU usage of the task-specific model can be recorded in real time to determine the computing load of the task-specific model at different processing stages. During data transmission bandwidth monitoring, the network bandwidth occupied by the task-specific model during data transmission can be monitored.

[0159] In addition, according to the processing steps and stages of the task-specific model, the percentage of completion of the task instruction in the current stage is determined, for example, for a complex task including multiple processing stages, by monitoring the completion of each stage, the overall completion degree is determined.

[0160] Further, the time from the start of processing of the task-specific model to the current time can be recorded, and according to the estimated total execution time, the remaining execution time of the current task instruction is calculated.

[0161] Further, state markers (such as "initialization", "processing", "completed", etc.) can be used to identify the current task progress state of the task-specific model.

[0162] In addition, according to the computing resource consumption and task progress state of the task-specific model, the demand of the remaining part of each sub-task in the task instruction for computing resources is determined. For example, for a task that has completed half of the processing flow and has high computing resource consumption, the demand for computing resources in the subsequent steps is estimated.

[0163] Further, according to factors such as the urgency, importance and remaining execution time of the task instruction, the resource allocation priority of each sub-task in the task instruction is determined, for example, urgent and important tasks will have higher priority and will be allocated with the remaining computing resources.

[0164] In addition, when it is monitored that there is a task-specific model that has completed processing of data features and general fusion features and there is remaining computing resources, the remaining computing resources are allocated to other task-specific models that have not completed processing. For example, if an image classification task-specific model has completed processing and there is remaining GPU resources, the remaining GPU resources can be allocated to another task-specific model that is performing a complex image generation task.

[0165] Further, according to the dynamic needs of each sub-task, the resource allocation strategy can be adjusted. For example, if there is a task-specific model that has insufficient computing resources during processing, the resource allocation of other sub-tasks can be temporarily adjusted to meet the computing resource demand of the sub-task.

[0166] The embodiment realizes real-time monitoring of the computing resource consumption and task progress state of each task-specific model, and allocates the remaining computing resources to other task-specific models that have not completed processing, thereby realizing resource optimization management of multi-modal general-special model collaborative reasoning, improving the utilization efficiency of resources, optimizing the execution efficiency of task instructions, and enhancing stability and reliability.

[0167] In order to effectively solve the problems of traditional technology in multi-modal data processing, such as semantic fusion difficulty, insufficient model flexibility, and failure to maximize the use of computing resources, and significantly improve the convenience of semantic fusion and the utilization rate of computing resources, the present application provides an embodiment of a device for implementing all or part of the multi-modal general-purpose model collaborative reasoning based on a dynamic routing mechanism, which is used to realize the multi-modal general-purpose model collaborative reasoning based on a dynamic routing mechanism, as shown in Figure 2 The device for implementing the multi-modal general-purpose model collaborative reasoning based on a dynamic routing mechanism specifically includes the following contents:

[0168] The receiving module 10 is configured to receive a task instruction and multi-modal data corresponding to the task instruction, determine a task type corresponding to the task instruction, extract data features corresponding to the multi-modal data, and fuse the data features based on a general-purpose model to obtain general-purpose fusion features. The task type includes a classification task, a generation task, and a retrieval task.

[0169] The mapping module 20 is configured to receive a preset processing mapping relationship, and determine a task-specific model, a modal processing flow, and fusion feature weights corresponding to the current task instruction based on the task type in the preset processing mapping relationship.

[0170] The processing module 30 is configured to send the data features and the general-purpose fusion features to the task-specific model through a routing node, so that the task-specific model processes the data features and the general-purpose fusion features based on the modal processing flow and the fusion feature weights to obtain specific fusion features, and processes the specific fusion features through a task execution component to obtain a task result corresponding to the task instruction.

[0171] From the above description, the application embodiment provides a multi-modal general-purpose model collaborative reasoning device based on a dynamic routing mechanism, which can receive task instructions and multi-modal data corresponding to the task instructions, determine the task type corresponding to the task instructions, and extract data features of the multi-modal data. The general-purpose model fuses the data features to generate general fusion features. According to a preset processing mapping relationship, a task-specific model, a modal processing flow, and a fusion feature weight corresponding to the current task type are matched. The data features and the general fusion features are transmitted to the task-specific model through the routing node. The task-specific model processes the data features and the general fusion features according to the modal processing flow and the fusion feature weight to obtain specific fusion features. The task execution component processes the specific fusion features to obtain the task result corresponding to the task instruction. The task-specific model and the fusion feature weight can be dynamically determined according to the task instruction. Multiple tasks can be processed simultaneously according to the same set of multi-modal data, which improves the reasoning efficiency and accuracy of different task classifications. The routing node allocates computing resources on demand, reduces redundant processing, and improves the efficiency and overall performance of multi-modal semantic fusion. This method can effectively solve the problems of traditional technology in multi-modal data processing, such as semantic fusion difficulty, insufficient model flexibility, and inefficient use of computing resources. It significantly improves the convenience of semantic fusion and the utilization rate of computing resources.

[0172] From the hardware level, in order to effectively solve the problems of traditional technology in multi-modal data processing, such as semantic fusion difficulty, insufficient model flexibility, and inefficient use of computing resources, significantly improve the convenience of semantic fusion, and improve the utilization rate of computing resources, the application provides an embodiment of an electronic device for implementing all or part of the contents of the multi-modal general-purpose model collaborative reasoning method based on a dynamic routing mechanism. The electronic device specifically includes the following contents:

[0173] A processor, a memory, a communication interface, and a bus; wherein the processor, the memory, the communication interface, and the bus complete mutual communication through the bus; the communication interface is used to realize information transmission between the multi-modal general-purpose model collaborative reasoning device based on a dynamic routing mechanism and the core business system, the user terminal, and the related database and other related devices; the logic controller can be a desktop computer, a tablet computer, and a mobile terminal, etc., and the embodiment is not limited thereto. In the embodiment, the logic controller can be implemented by referring to the embodiment of the multi-modal general-purpose model collaborative reasoning method based on a dynamic routing mechanism, and the embodiment of the multi-modal general-purpose model collaborative reasoning device based on a dynamic routing mechanism. The contents are incorporated herein, and the repeated parts will not be described again.

[0174] It can be understood that the user terminal can include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device can include smart glasses, a smart watch, a smart bracelet, etc.

[0175] In actual application, part of the multi-modal general-special model collaborative reasoning method based on the dynamic routing mechanism can be executed on the electronic device as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capability of the client device and the limitation of the user's use scene. The present application does not limit this. If all operations are completed in the client device, the client device can also include a processor.

[0176] The above-mentioned client device can have a communication module (i.e., a communication unit) and can be communicatively connected with a remote server to realize data transmission with the server. The server can include a server of the task scheduling center side, and in other implementation scenarios, it can also include a server of an intermediate platform, such as a server of a third-party server platform communicatively connected with the server of the task scheduling center. The server can include a single computer device, or a server cluster composed of multiple servers, or a server structure of a distributed device.

[0177] Figure 3 A schematic block diagram of the system structure of the electronic device 9600 of an embodiment of the present application is shown in FIG. 9. As shown in the figure, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that the structure shown in the figure is exemplary; other types of structures can also be used to supplement or replace the structure to realize telecommunication functions or other functions. Figure 3 Figure 3 The structure shown in the figure is exemplary; other types of structures can also be used to supplement or replace the structure to realize telecommunication functions or other functions.

[0178] In an embodiment, the multi-modal general-special model collaborative reasoning method based on the dynamic routing mechanism can be integrated into the central processor 9100. The central processor 9100 can be configured to control as follows:

[0179] Step S101: receiving a task instruction and multi-modal data corresponding to the task instruction, determining a task type corresponding to the task instruction, extracting data features corresponding to the multi-modal data, and fusing the data features based on a general task model to obtain general fusion features, the task type including a classification task, a generation task and a retrieval task;

[0180] ​Step S102: Receive the preset processing mapping relationship, and determine the task-specific model, modal processing flow and fusion feature weights corresponding to the current task instruction based on the task type in the preset processing mapping relationship;

[0181] Step S103: Send the data features and general fusion features to the task-specific model through the routing node, so that the task-specific model processes the data features and general fusion features based on the modal processing flow and fusion feature weights to obtain the special fusion features, and processes the special fusion features through the task execution component to obtain the task result corresponding to the task instruction.

[0182] As described above, the electronic device provided in this application embodiment innovatively receives task instructions and corresponding multimodal data, determines the task type corresponding to the task instructions, extracts data features from the multimodal data, generates general fusion features by fusing data features through a general task model, matches the task-specific model, modal processing flow, and fusion feature weights of the current task type according to a preset processing mapping relationship, and transmits the data features and general fusion features to the task-specific model through a routing node. The task-specific model processes the data features and general fusion features according to the modal processing flow and fusion feature weights to obtain the specific fusion features. The task execution component processes the specific fusion features to obtain the task result corresponding to the task instructions. It can dynamically determine the task-specific model and fusion feature weights according to the task instructions, and can also process multiple tasks simultaneously based on the same set of multimodal data, improving the inference efficiency and accuracy corresponding to different task categories. By allocating computing resources on demand through routing nodes, redundant processing is reduced, improving the efficiency and overall performance of multimodal semantic fusion. This method can effectively address the shortcomings of traditional techniques in multimodal data processing, such as difficulties in semantic fusion, insufficient model flexibility, and failure to maximize the utilization of computing resources. It significantly improves the convenience of semantic fusion and enhances the utilization rate of computing resources.

[0183] In another embodiment, a multimodal communication-specific model collaborative reasoning device based on a dynamic routing mechanism can be configured separately from the central processing unit 9100. For example, a multimodal communication-specific model collaborative reasoning device based on a dynamic routing mechanism can be configured as a chip connected to the central processing unit 9100, and the function of the multimodal communication-specific model collaborative reasoning method based on the dynamic routing mechanism can be realized through the control of the central processing unit.

[0184] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3Components not shown in the figure can be referred to the prior art.

[0185] As shown in FIG. 9, the central processing unit 9100, which is also sometimes referred to as a controller or operation control, can include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of the various components of the electronic device 9600. Figure 3

[0186] The memory 9140, for example, can be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. Information relating to failures described above can be stored, in addition to programs for executing the information. The central processing unit 9100 can execute the programs stored in the memory 9140 to perform information storage or processing, etc.

[0187] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0188] The memory 9140 can be a solid state memory such as a read only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROM, etc. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 can include an application / function storage section 9142 for storing application programs and function programs or for storing a flow for performing the operation of the electronic device 9600 by the central processing unit 9100.

[0189] The memory 9140 can also include a data storage section 9143 for storing data such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. A driver storage section 9144 of the memory 9140 can include various drivers of the electronic device for communication functions and / or for performing other functions of the electronic device such as a messaging application, a contact application, etc.

[0190] ​The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of a conventional mobile communication terminal.

[0191] Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc., can be provided in the same electronic device. The communication module 9110 (transmitter / receiver) is also coupled to the speaker 9131 and the microphone 9132 via the audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing the usual telecommunication functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording on the local device via the microphone 9132 and playing stored sounds on the local device via the speaker 9131.

[0192] The embodiment of the present application also provides a computer readable storage medium capable of implementing all steps of the multi-modal general-purpose model collaborative reasoning method based on a dynamic routing mechanism with the execution subject being a server or a client in the above-mentioned embodiment, and the computer program is stored on the computer readable storage medium. When the processor executes the computer program, all steps of the multi-modal general-purpose model collaborative reasoning method based on a dynamic routing mechanism with the execution subject being a server or a client in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0193] Step S101: receiving a task instruction and multi-modal data corresponding to the task instruction, determining a task type corresponding to the task instruction, extracting data features corresponding to the multi-modal data, and fusing the data features based on a task general-purpose model to obtain general-purpose fusion features, the task type including a classification task, a generation task and a retrieval task;

[0194] Step S102: receiving a preset processing mapping relationship, determining a task-specific model, a modal processing flow and a fusion feature weight corresponding to the current task instruction based on the task type in the preset processing mapping relationship;

[0195] Step S103: sending the data features and the general-purpose fusion features to the task-specific model through a routing node, so that the task-specific model processes the data features and the general-purpose fusion features based on the modal processing flow and the fusion feature weight to obtain specific fusion features, and processes the specific fusion features through a task execution component to obtain a task result corresponding to the task instruction.

[0196] From the above description, the computer readable storage medium provided by the embodiments of the application can receive task instructions and multi-modal data corresponding to the task instructions innovatively, determine the task type corresponding to the task instructions, extract data features of the multi-modal data, generate general fusion features by fusing the data features through a task general model, match the task-specific model, the modal processing flow and the fusion feature weight of the current task type according to the preset processing mapping relationship, transmit the data features and the general fusion features to the task-specific model through the routing node, make the task-specific model process the data features and the general fusion features according to the modal processing flow and the fusion feature weight, obtain the special fusion features, and process the special fusion features through the task execution component to obtain the task result corresponding to the task instruction. The task-specific model and the fusion feature weight can be dynamically determined according to the task instruction, and multiple tasks can be processed simultaneously according to the same set of multi-modal data, which improves the inference efficiency and accuracy of different task classifications, and improves the efficiency and overall performance of multi-modal semantic fusion by allocating computing resources on demand through the routing node, reducing redundant processing. This method can effectively solve the problems of traditional technologies, such as difficulty in semantic fusion when processing multi-modal data, insufficient model flexibility, and failure to maximize the use of computing resources, significantly improving the convenience of semantic fusion and the utilization rate of computing resources.

[0197] The embodiments of the application also provide a computer program product capable of implementing all steps of the multi-modal general-special model collaborative inference method based on the dynamic routing mechanism in which the execution subject in the above embodiments is a server or a client. The computer program / instructions are executed by the processor to implement the steps of the multi-modal general-special model collaborative inference method based on the dynamic routing mechanism, for example, the computer program / instructions implement the following steps:

[0198] Step S101: receiving task instructions and multi-modal data corresponding to the task instructions, determining the task type corresponding to the task instructions, extracting data features corresponding to the multi-modal data, and fusing the data features based on a task general model to obtain general fusion features, the task type including a classification task, a generation task and a retrieval task;

[0199] Step S102: receiving a preset processing mapping relationship, determining a task-specific model, a modal processing flow and a fusion feature weight corresponding to the current task instruction based on the task type in the preset processing mapping relationship;

[0200] Step S103: sending the data features and the general fusion features to the task-specific model through the routing node, so that the task-specific model processes the data features and the general fusion features based on the modal processing flow and the fusion feature weight to obtain special fusion features, and processes the special fusion features through the task execution component to obtain the task result corresponding to the task instruction.

[0201] From the above description, the computer program product provided by the embodiment of the application can receive a task instruction and multi-modal data corresponding to the task instruction, determine a task type corresponding to the task instruction, and extract data features of the multi-modal data. The general fusion features are generated by fusing the data features through a task general model. The task-specific model, the modal processing flow, and the fusion feature weight corresponding to the current task type are matched according to the preset processing mapping relationship. The data features and the general fusion features are transmitted to the task-specific model through the routing node, so that the task-specific model processes the data features and the general fusion features according to the modal processing flow and the fusion feature weight, and obtains specific fusion features. The specific fusion features are processed by the task execution component to obtain a task result corresponding to the task instruction. The task-specific model and the fusion feature weight can be dynamically determined according to the task instruction. Multiple tasks can be processed simultaneously according to the same set of multi-modal data. The inference efficiency and accuracy corresponding to different task classifications are improved. The computing resources are allocated on demand through the routing node, the redundant processing is reduced, and the efficiency and overall performance of multi-modal semantic fusion are improved. The method can effectively solve the problems of the traditional technology, such as difficulty in semantic fusion during multi-modal data processing, insufficient model flexibility, and failure to maximize the use of computing resources, significantly improves the convenience of semantic fusion, and improves the utilization rate of computing resources.

[0202] Those skilled in the art will understand that embodiments of the application can be provided as methods, apparatus, or computer program products. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0203] The application is described with reference to flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0204] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or multiple blocks.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or multiple blocks.

[0206] The principles and implementations of the present application are described in the specific embodiments, the above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A collaborative reasoning method for a multimodal communication model based on a dynamic routing mechanism, characterized in that, The method includes: The system receives task instructions and corresponding multimodal data, determines the task type corresponding to the task instructions, extracts data features corresponding to the multimodal data, and fuses the data features based on a general task model to obtain general fused features. The task types include classification tasks, generation tasks, and retrieval tasks. The step of fusing the data features based on the general task model to obtain general fused features includes: mapping the data features to a preset semantic space in the general task model to obtain mapped data features; determining the similarity between the mapped data features; assigning general feature weights to the mapped data features based on the similarity, where the similarity is positively correlated with the feature weights; and fusing the data features through the general task model based on the feature weights to obtain general fused features. The method includes receiving a preset processing mapping relationship, and determining the task-specific model, modal processing flow, and fusion feature weights corresponding to the current task instruction based on the task type within the preset processing mapping relationship. Before receiving the preset processing mapping relationship, the method further includes constructing a preset processing mapping relationship library, which includes combinations of task-specific models, modal processing flows, and fusion feature weights corresponding to each task type. The step of determining the task-specific model, modal processing flow, and fusion feature weights corresponding to the current task instruction based on the task type within the preset processing mapping relationship includes determining, based on the task type, the task-specific model, modal processing flow, and fusion feature weights matching the current task type from the preset processing mapping relationship library. The data features and the general fusion features are sent to the task-specific model through the routing node, so that the task-specific model processes the data features and the general fusion features based on the modal processing flow and the fusion feature weights to obtain the specific fusion features. The task execution component then processes the specific fusion features to obtain the task result corresponding to the task instruction.

2. The method according to claim 1, characterized in that, The process of processing the dedicated fusion features through the task execution component to obtain the task result corresponding to the task instruction includes: When the task type corresponding to the task instruction is the classified task, the special fusion feature is processed by a classification algorithm to obtain the classification label corresponding to the task instruction; When the task type corresponding to the task instruction is the generation task, the special fusion feature is processed by the generation model to generate at least one of the text generation result, image generation result and audio generation result corresponding to the task instruction; When the task type corresponding to the task instruction is the retrieval task, the special fusion feature is processed by the retrieval algorithm to obtain the retrieval result corresponding to the task instruction.

3. The method according to claim 1, characterized in that, After determining the task-specific model, modal processing flow, and fusion feature weights corresponding to the current task instruction based on the task type in the preset processing mapping relationship, the method further includes: The complexity level of the task instruction is determined, wherein the complexity level is determined based on the data size of the multimodal data and the task type; The computational resources required to compute the multimodal data during the process of completing the task instruction based on the multimodal data are determined. Based on the complexity level and the computational resources, a resource allocation strategy corresponding to the current task instruction is determined. The computational resources include memory resources, video memory resources, and central processing unit computing power requirements. Sending the data features and the general fusion features to the task-specific model via a routing node includes: Based on the resource allocation strategy, the data features and the general fusion features are sent to the corresponding task-specific model through the routing node.

4. The method according to claim 1, characterized in that, After sending the data features and the general fusion features to the task-specific model via the routing node, the method further includes: Real-time monitoring of the execution performance metrics of each task-specific model, including accuracy, recall, execution time, and consumption of computing resources; Based on the execution performance indicators, the processing parameters in the modal processing flow of the preset processing mapping relationship are dynamically adjusted.

5. The method according to claim 4, characterized in that, The step of dynamically adjusting the processing parameters in the modal processing flow within the preset processing mapping relationship based on the execution performance indicators includes: The computational resource consumption of each task-specific model during the processing of the data features and the general fusion features is monitored in real time, and the task progress status corresponding to the task instruction is monitored in real time. When it is detected that a task-specific model has completed processing of the data features and the general fusion features and has remaining computing resources, while other task-specific models have not completed processing of the data features and the general fusion features, the remaining computing resources are allocated to the other task-specific models.

6. A multimodal communication-specific model collaborative reasoning device based on a dynamic routing mechanism, characterized in that, The device includes: A receiving module is configured to receive task instructions and corresponding multimodal data, determine the task type corresponding to the task instructions, extract data features corresponding to the multimodal data, and fuse the data features based on a general task model to obtain general fused features. The task types include classification tasks, generation tasks, and retrieval tasks. The step of fusing the data features based on the general task model to obtain general fused features includes: mapping the data features to a preset semantic space in the general task model to obtain mapped data features; determining the similarity between the mapped data features; assigning general feature weights to the mapped data features based on the similarity, wherein the similarity is positively correlated with the feature weights; and fusing the data features through the general task model based on the feature weights to obtain general fused features. A mapping module is used to receive a preset processing mapping relationship, and determine the task-specific model, modal processing flow, and fusion feature weights corresponding to the current task instruction based on the task type in the preset processing mapping relationship; before receiving the preset processing mapping relationship, the module further includes: constructing a preset processing mapping relationship library, the preset processing mapping relationship library including combinations of task-specific models, modal processing flows, and fusion feature weights corresponding to each task type; the step of determining the task-specific model, modal processing flow, and fusion feature weights corresponding to the current task instruction based on the task type in the preset processing mapping relationship includes: determining the task-specific model, modal processing flow, and fusion feature weights matching the current task type from the preset processing mapping relationship library based on the task type; The processing module is used to send the data features and the general fusion features to the task-specific model through a routing node, so that the task-specific model processes the data features and the general fusion features based on the modal processing flow and the fusion feature weights to obtain the specific fusion features, and processes the specific fusion features through the task execution component to obtain the task result corresponding to the task instruction.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal communication model collaborative reasoning method based on dynamic routing mechanism as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal communication model collaborative reasoning method based on dynamic routing mechanism as described in any one of claims 1 to 5.

Images