Multi-modal data processing method and system, computer equipment and readable storage medium

Abstract

The invention discloses a multi-modal data processing method and system, computer equipment and a readable storage medium, which can realize deep association and complementarity mining of multi-modal information and improve the accuracy and robustness of multi-modal understanding. The method comprises the following steps: an environment sensing module adjusts an environment sensing strategy according to feedback information transmitted by a self-adaptive decision module, and acquires multi-modal data according to the environment sensing strategy; the multi-modal encoding module encodes the multi-modal data into multi-modal feature vectors of the same dimension; a cross-modal fusion module fuses the multi-modal feature vectors to obtain fusion features; the self-adaptive decision-making module selects a decision-making network matched with the task type from a predefined network library according to the task type of the current decision-making task, inputs the fusion features into the decision-making network, and generates feedback information according to the decision-making process of the decision-making network; and the meta-learning controller evaluates the system performance of the current multi-modal data processing system and adjusts system parameters according to an evaluation result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal data processing method, system, computer device, and readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, AI agents, as computing systems that can perceive the environment, make decisions and perform tasks, have been widely used in fields such as medicine, manufacturing, and customer service. Among related technologies, mainstream AI agent systems mostly adopt a multimodal fusion architecture based on deep learning. For example, a pre-trained BERT model is used to process text, a CNN model is used to extract image features, and then multimodal information is fused by feature splicing or weighted averaging. Although such systems can achieve basic multimodal data processing, their fusion mechanism only stays at the shallow feature integration level, and each modal processing module is optimized independently, lacking end-to-end collaborative training. In addition, existing systems usually use fixed parameter models, which make it difficult to dynamically adjust processing strategies according to environmental changes, resulting in system performance degradation when facing data distribution shifts or new tasks. Summary of the Invention

[0003] In view of this, the present application provides a multimodal data processing method, system, computer device and readable storage medium, the main purpose of which is to solve the problem that some systems usually adopt fixed parameter models and it is difficult to dynamically adjust the processing strategy according to environmental changes, resulting in performance degradation of the system when facing data distribution offset or new tasks.

[0004] According to a first aspect of the present application, a multimodal data processing method is provided, the method comprising:

[0005] The environmental perception module adjusts the environmental perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environmental perception strategy, wherein the feedback information includes but is not limited to the modality importance score, the data quality assessment result and the information missing mark;

[0006] The multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension;

[0007] The cross-modal fusion module adopts a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature. The fusion weight is adjusted according to the task type of the current decision task and the input data characteristics;

[0008] The adaptive decision module selects a decision network that matches the task type from a predefined network library according to the task type of the current decision task, inputs the fusion feature into the decision network, generates feedback information according to the decision process of the decision network, and transmits the feedback information to the environment perception module through a reverse communication channel;

[0009] The meta-learning controller evaluates the system performance of the current multimodal data processing system and adjusts system parameters according to the evaluation results. The system parameters include hyperparameters and network structures of designated modules. The designated modules include the multimodal encoding module, the cross-modal fusion module and the adaptive decision module.

[0010] Optionally, the environment perception module adjusts the environment perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environment perception strategy, including:

[0011] The environment perception module receives the feedback information, associates the feedback information with corresponding multimodal data according to the timestamp and task identifier carried in the feedback information, and stores the associated feedback information in a knowledge base;

[0012] The environmental perception module adjusts the sampling parameters indicated by the data acquisition strategy according to the modal importance score in the feedback information, and adjusts the re-acquisition trigger threshold or the sampling parameters indicated by the data acquisition strategy according to the data quality assessment result in the feedback information, and adjusts the attention allocation result indicated by the attention allocation strategy according to the information missing flag in the feedback information, and adjusts the preprocessing rules indicated in the data preprocessing strategy according to the data quality assessment result in the feedback information, wherein the sampling parameters include but are not limited to sampling frequency and sampling accuracy, and the environmental perception strategy includes the data acquisition strategy, the attention allocation strategy, and the data preprocessing strategy;

[0013] The environmental perception module obtains the multimodal data through multiple data acquisition sensors or API interfaces according to the sampling parameters indicated by the data acquisition strategy and the attention allocation results indicated by the attention allocation strategy, and preprocesses the multimodal data according to the preprocessing rules indicated by the data preprocessing strategy, and inputs the preprocessed multimodal data into the multimodal encoding module.

[0014] Optionally, the multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension, including:

[0015] The multimodal encoding module determines a feature extraction network corresponding to each modal data, wherein the text modal data uses a Transformer encoder, the image modal data uses a visual Transformer encoder or a CNN network, and the audio modal data uses a waveform encoder;

[0016] The multimodal encoding module adjusts the encoding depth and encoding width corresponding to each feature extraction network according to the signal-to-noise ratio of the input data and the task complexity required by the task, and extracts data features corresponding to each modal data based on the adjusted feature extraction network;

[0017] The multimodal encoding module maps the data features corresponding to each modality data to a shared semantic space based on contrastive learning technology to obtain a multimodal feature vector of the same dimension.

[0018] Optionally, the cross-modal fusion module adopts a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature, including:

[0019] The cross-modal fusion module uses a multi-head cross-modal attention mechanism to calculate the association weight between each modal feature vector to obtain an initial fusion weight, and constructs a task feature vector according to the task type, maps the task feature vector to a task-related weight through a deep neural network, and determines the quality score and complementarity score corresponding to the multimodal feature vector, and generates a data feature-related weight based on the quality score and the complementarity score, wherein the quality score is calculated based on the signal-to-noise ratio and completeness of each modal feature vector, and the complementarity score is calculated based on the mutual information between the feature vectors of different modalities. The task type is obtained by identifying the user instruction;

[0020] The cross-modal fusion module performs a weighted summation on the initial fusion weight, the task-related weight, and the data feature-related weight based on the balance coefficient to obtain a comprehensive weight, constructs a task loss function according to the task performance indicator, continuously optimizes the balance coefficient using the task loss function and a weight regularization term through a backpropagation algorithm to obtain an optimized comprehensive weight, and normalizes the optimized comprehensive weight using a normalization function to obtain the fusion weight;

[0021] The cross-modal fusion module performs weighted fusion on the multimodal feature vectors based on the fusion weights to obtain the fusion features.

[0022] Optionally, the adaptive decision module selects a decision network matching the task type from a predefined network library according to the task type of the current decision task, inputs the fusion feature into the decision network, and generates feedback information according to the decision process of the decision network, including:

[0023] The adaptive decision module parses the user instruction and determines the intention encoding vector of the user instruction through a pre-trained language model, performs similarity matching between the intention encoding vector and the task feature vector in the preset task library, determines the task type based on the matching result, and loads the adapted decision network architecture from the predefined network library based on the task type;

[0024] The adaptive decision module inputs the fusion features into the decision network, records the decision process, and extracts the attention weights of different modal feature channels in the decision network based on the decision process to construct a modal contribution matrix to obtain the modal importance score, and determines the signal-to-noise ratio and integrity of the input data based on the decision process, generates the data quality assessment result according to the signal-to-noise ratio and the integrity, and detects underused key information based on the decision process to generate the information missing identifier, and encapsulates the modal importance score, the data quality assessment result and the information missing identifier into structured feedback information, wherein the modal importance score is used to indicate the contribution ratio of each modal data to the decision result.

[0025] Optionally, the method further includes:

[0026] The adaptive decision module records the decision results output by the decision network, calculates the variance of the decision results based on Monte Carlo Dropout sampling, generates a confidence score, associates the confidence score with the decision results, and records explanation information, associates the explanation information with the decision results, the explanation information includes a global explanation, a modal explanation, and a local explanation, wherein the global explanation includes a task type, a decision goal, and an influencing factor, the modal explanation includes a modal importance score, and the local explanation includes a reasoning process of a specified decision point in the decision network;

[0027] The adaptive decision module converts the decision result, the confidence score and the explanation information into a textual explanation using a pre-trained natural language generation model, and stores the textual explanation in a knowledge base;

[0028] The meta-learning controller evaluates the system performance of the current multimodal data processing system, queries the historical task model configuration in the knowledge base according to the evaluation result, and adjusts the current system parameters according to the historical task model configuration.

[0029] According to a second aspect of the present application, a multimodal data processing system is provided, the system comprising:

[0030] The environmental perception module adjusts the environmental perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environmental perception strategy, wherein the feedback information includes but is not limited to the modality importance score, the data quality assessment result and the information missing mark;

[0031] The multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension;

[0032] The cross-modal fusion module uses a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature. The fusion weight is adjusted according to the task type of the current decision task and the input data characteristics;

[0033] The adaptive decision module selects a decision network that matches the task type from a predefined network library according to the task type of the current decision task, inputs the fusion feature into the decision network, generates feedback information according to the decision process of the decision network, and transmits the feedback information to the environment perception module through a reverse communication channel;

[0034] The meta-learning controller evaluates the system performance of the current multimodal data processing system and adjusts system parameters based on the evaluation results. The system parameters include hyperparameters and network structures of designated modules. The designated modules include the multimodal encoding module, the cross-modal fusion module, and the adaptive decision module.

[0035] According to a third aspect of the present application, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0036] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0037] By means of the above technical solution, the present application provides a multimodal data processing method, system, computer device and readable storage medium. The embodiment of the present application dynamically adjusts the fusion weight through a multi-head cross-modal attention mechanism to achieve deep correlation and complementary mining of multimodal information, thereby improving the accuracy and robustness of multimodal understanding. And through the cross-task parameter optimization of the meta-learning controller, the system can dynamically adapt to new scenarios and reduce performance fluctuations caused by environmental changes. In addition, the data collection and processing process is continuously optimized through a closed-loop feedback mechanism to achieve stable operation and continuous evolution of the system in complex scenarios.

[0038] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0040] Figure 1 A schematic diagram of a multimodal data processing method provided in an embodiment of the present application is shown;

[0041] Figure 2 A schematic diagram of the structure of a multimodal data processing system provided in an embodiment of the present application is shown;

[0042] Figure 3 A schematic diagram of the workflow of a multimodal encoding module of a multimodal data processing method provided in an embodiment of the present application is shown;

[0043] Figure 4 A schematic diagram of the cross-modal fusion module workflow of a multimodal data processing method provided in an embodiment of the present application is shown;

[0044] Figure 5 A schematic diagram of a meta-learning controller workflow of a multimodal data processing method provided in an embodiment of the present application is shown;

[0045] Figure 6 A schematic diagram of the structure of another multimodal data processing system provided in an embodiment of the present application is shown;

[0046] Figure 7 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0048] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0050] Those skilled in the art will appreciate that the term "terminal" as used herein includes both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of performing two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices with single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) devices that may combine voice, data processing, fax, and / or data communication capabilities; PDAs (Personal Digital Assistants) that may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices that have and / or include a radio frequency receiver. As used herein, a "terminal" may be portable, transportable, installed in a vehicle (air, sea, and / or land), or adapted and / or configured to operate locally, and / or in a distributed manner, at any other location on Earth and / or in space. As used herein, a "terminal" may also be a communication terminal, an Internet access terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a device such as a smart TV or a set-top box.

[0051] The embodiments of the present application are applicable to multimodal data processing. Multimodal data refers to data sets containing multiple different types of information. These different data modalities can describe real-world objects, events, or phenomena from multiple dimensions and perspectives, enabling computers to understand and process information more comprehensively and deeply. Common modalities include visual, auditory, and textual modalities. Visual modalities include data forms such as images and videos. Images are instantaneous captures of static scenes, containing rich information such as the object's appearance, shape, and color. For example, medical images such as X-rays and CT scans can help doctors diagnose diseases. Videos are a series of continuous image frames. In addition to containing image information, they also add dynamic information along the temporal dimension. They are widely used in fields such as security monitoring, video conferencing, and film and television production. Auditory modalities primarily exist in the form of audio and speech. Audio encompasses a variety of sound signals, such as music, natural environmental sounds, and the sounds of machinery. Speech is the sound produced by humans through their vocal organs. It contains semantic and emotional information and plays an important role in fields such as intelligent voice assistants, voice translation, and telephone customer service. Textual modality is a form of data based on language and text, including news articles, books, social media posts, product manuals, and more. Text data is highly abstract and semantic, capable of conveying precise information and knowledge. In natural language processing, text data can be used for tasks such as information retrieval, text classification, sentiment analysis, and machine translation. Multimodal data exhibits both complementarity and redundancy. Specifically, complementarity means that data from different modalities can complement each other, providing more complete information. For example, in video surveillance, the visual modality can reveal the movements and context of people, while the audio modality can hear the conversation. The combination of the two provides a more accurate understanding of the course of events. Redundancy means that data from different modalities may contain overlapping information. For example, in a news report, both the text description and the associated image may convey key information about the event. Leveraging this redundancy can improve data reliability and stability. When data from one modality is missing or corrupted, data from other modalities can still provide relevant information.

[0052] The present application provides a multimodal data processing method, such as Figure 1 As shown, the method includes:

[0053] 101. The environmental perception module adjusts the environmental perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environmental perception strategy, wherein the feedback information includes but is not limited to the modal importance score, the data quality assessment result and the information missing mark.

[0054] The multimodal data processing method in the embodiment of the present application is applicable to a multimodal data processing system, such as Figure 2As shown, it includes an environment perception module, a multimodal encoding module, a cross-modal fusion module (cross-modal attention fusion module), an adaptive decision module, a meta-learning controller and a knowledge base.

[0055] In this step, the environmental perception module is responsible for acquiring multimodal data from the external environment. Multimodal data includes but is not limited to text, images, audio, video and structured data. During the actual operation process, the module acquires multimodal data through a variety of data acquisition sensors (such as cameras, microphones, etc.) or API interfaces according to the sampling parameters indicated by the data acquisition strategy and the attention allocation results indicated by the attention allocation strategy, and gives priority to acquiring data in high-value areas (such as image data, video data, etc.). Furthermore, the multimodal data is preprocessed according to the preprocessing rules indicated by the data preprocessing strategy, and the preprocessed multimodal data is input into the multimodal encoding module. It is understandable that the preprocessing rules may include noise reduction algorithms, standardization operations, and data alignment operations, etc. Relevant technical personnel can pre-set the preprocessing rules according to actual needs. This application does not specifically limit the content of the preprocessing rules.

[0056] It is understood that in order to dynamically adjust the sampling frequency, accuracy, and preprocessing rules, enabling the system to quickly adapt to environmental changes and avoid perception failures caused by fixed parameters, the environmental perception module also needs to receive feedback from the adaptive decision-making module and optimize the environmental perception strategy based on the feedback information. The environmental perception strategy includes a data acquisition strategy, an attention allocation strategy, and a data preprocessing strategy. Specifically, the environmental perception module first associates the feedback information with the corresponding multimodal data based on the timestamp and task identifier carried in the feedback information, and stores the associated feedback information in a knowledge base to provide data support for subsequent strategy optimization and model iteration. Furthermore, the environmental perception module adjusts the sampling parameters indicated by the data acquisition strategy based on the modal importance score in the feedback information. The sampling parameters include but are not limited to the sampling frequency and sampling accuracy. For example, when the modal importance score is greater than a preset threshold, the sampling frequency of the corresponding sensor is increased (e.g., from 30Hz to 60Hz). Based on the data quality assessment results in the feedback information, the re-acquisition trigger threshold or sampling parameters indicated by the data acquisition strategy are adjusted. For example, if the missing rate in the quality assessment result is greater than the re-acquisition trigger threshold, the data re-acquisition protocol is initiated. According to the information missing flag in the feedback information, the attention allocation result indicated by the attention allocation strategy is adjusted. According to the data quality assessment result in the feedback information, the preprocessing rules indicated by the data preprocessing strategy are adjusted.

[0057] Furthermore, the environmental perception module acquires multimodal data through various data acquisition sensors or APIs based on the sampling parameters indicated by the data acquisition strategy and the attention allocation results indicated by the attention allocation strategy. The multimodal data is then preprocessed according to the preprocessing rules indicated by the data preprocessing strategy and input into the multimodal encoding module.

[0058] 102. The multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension.

[0059] In the embodiments of this application, Figure 3 As shown in the figure, the multimodal encoding module first determines the corresponding feature extraction network for each modal data. Among them, the text modal data uses the Transformer encoder, the image modal data uses the visual Transformer encoder or CNN network, and the audio modal data uses the waveform encoder. Transformer is a neural network architecture based on the self-attention mechanism. It was originally used for natural language processing tasks and is now widely used in data processing of various modalities. CNN (Convolutional Neural Network) is a deep learning architecture specifically used to process data with grid structure (such as images). Furthermore, the multimodal encoding module adjusts the encoding depth and encoding width corresponding to each feature extraction network according to the signal-to-noise ratio of the input data and the task complexity required by the task. For example, for simple tasks with a classification number of less than or equal to 3, a 3-layer Transformer encoding is used, and for complex tasks with a classification number of greater than 3, a 6-layer deep encoding is used. Next, based on the adjusted feature extraction network, the data features corresponding to each modal data are extracted. Finally, based on the contrastive learning technology, the multimodal encoding module aligns each modal data X1, X2,…, X1 into a shared semantic space constrained by the contrastive learning loss function. n Corresponding data features, the contrastive learning loss function can guide the feature extraction network to learn more discriminative feature representations, so that data of different modalities have better alignment effects in the shared semantic space. Specifically, the module can be obtained by the following formula Z i =W i X i +b i The data features corresponding to each modal data are mapped to the shared semantic space respectively to obtain a multimodal feature vector of the same dimension d.

[0060] In summary, the multimodal encoding module achieves effective encoding of data of different modalities through a series of operations such as determining the feature extraction network, adjusting the encoding depth and width, extracting data features, and mapping to a shared semantic space, providing important support for the multimodal data processing of the entire system.

[0061] 103. The cross-modal fusion module adopts a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature. The fusion weight is adjusted according to the task type of the current decision task and the input data characteristics.

[0062] In the embodiment of the present application, the cross-modal fusion module is responsible for capturing the deep semantic associations between different modalities and achieving effective fusion of information. Cross-modal attention is a neural network mechanism used to calculate the correlation weights between different modal features and achieve information interaction and fusion between modalities. Specifically, Figure 4 As shown, the cross-modal fusion module uses a multi-head cross-modal attention mechanism to calculate the association weights between each modality feature vector to obtain the initial fusion weight W init , where W init =[W1,W2,...,W n ], each W i Indicates the importance of the i-th mode. Next, construct the task feature vector t according to the task type T T , including key indicators such as task complexity and modal dependency. The task feature vector is mapped to task-related weights through a deep neural network, and the mapping function is W task =f task (t T ). Further, determine the quality score q corresponding to the multimodal feature vector i and complementarity score c ij , according to the mass fraction q i and complementarity score c ij Generate data feature related weights W data =f data ({q i},{c ij}), where the mass fraction q i It is calculated based on the signal-to-noise ratio and completeness of each modal eigenvector, and the complementarity score c ij It is calculated based on the mutual information between the modal feature vector i and the modal feature vector j. Furthermore, the cross-modal fusion module adjusts the initial fusion weight W based on the balance coefficients α, β, and γ. init , task-related weight W task and the weight W associated with the data features data Perform weighted summation to obtain the comprehensive weight W final =softmax(α·W task +β·W data +γ·W init). It should be noted that the cross-modal fusion module also needs to construct a task loss function based on the task performance indicators, and continuously optimize the balance coefficient using the task loss function and weight regularization term through the back propagation algorithm to obtain the optimized comprehensive weight. Among them, the loss function of the comprehensive weight optimization is L weight =L tast +λ·R(W final ), where L tast is the task loss function, R(W final ) is the weight regularization term, and λ is an adjustable parameter. The cross-modal fusion module then uses a normalization function to normalize the optimized comprehensive weights to obtain fusion weights. Finally, the cross-modal fusion module performs a weighted fusion of the multimodal feature vectors based on the fusion weights to obtain fused features.

[0063] It is understandable that in actual operation, in addition to the cross-modal attention fusion mechanism, graph neural network fusion, dynamic routing fusion, and adversarial fusion can also be used to generate fused features. Specifically, graph neural network fusion treats different modal features as nodes of a graph, capturing the relationship between modalities through a graph convolutional network, achieving information fusion and obtaining fused features. Dynamic routing fusion draws on the concept of capsule networks, using a capsule network dynamic routing algorithm to iteratively adjust the modal combination weights, and then uses this combination weight to fuse multimodal feature vectors to obtain fused features. Adversarial fusion uses a generative adversarial network framework to train a modality fuser to generate fused features.

[0064] 104. The adaptive decision module selects a decision network that matches the task type from the predefined network library according to the task type of the current decision task, inputs the fusion features into the decision network, generates feedback information according to the decision process of the decision network, and transmits the feedback information to the environment perception module through the reverse communication channel.

[0065] In an embodiment of the present application, the adaptive decision-making module first parses the user's input command. Once the user issues a command, the module rapidly processes it and converts it into an intent encoding vector using a pre-trained language model. This pre-trained language model, trained on a large amount of text data, possesses semantic understanding capabilities and accurately captures the key information and semantic intent within the user's command, thereby converting the command into a vector representation. The adaptive decision-making module then performs a similarity match between the resulting intent encoding vector and the task feature vectors in a pre-defined task library. The pre-defined task library stores feature vectors for various common tasks. These vectors are abstract representations of different tasks and reflect their essential characteristics. By calculating the similarity between the intent encoding vector and the task feature vector, the module identifies the task that best matches the user's command. Based on the matching results, the module accurately determines the task type corresponding to the user's command. After determining the task type, the adaptive decision-making module loads an adapted decision network architecture from a pre-defined network library based on the task type. The pre-defined network library contains a variety of decision network architectures designed for different task types. By loading adapted decision network architectures, the system can better address the user's specific needs and improve decision-making accuracy and efficiency.

[0066] After loading the decision network architecture, the adaptive decision module inputs the fused features into the decision network for decision analysis. During the decision-making process, the module records every step and operation in detail for subsequent decision analysis. Based on this recorded decision process, the module extracts the attention weights for different modal feature channels in the decision network. Attention weights reflect the importance of each modal feature in the decision-making process. By analyzing these weights, the module constructs a modal contribution matrix. The modal contribution matrix intuitively displays the contribution of each modal data to the decision outcome. Based on this matrix, a modal importance score can be calculated, which clearly indicates the relative importance of each modal data in the decision. Furthermore, the adaptive decision module assesses the signal-to-noise ratio and integrity of the input data based on the decision-making process. The signal-to-noise ratio reflects the signal-to-noise ratio in the data, while integrity indicates whether the data is complete and free of omissions. By evaluating these two metrics, the module generates a data quality assessment result, which helps the system understand the quality of the input data and provides a reference for subsequent data processing and decision-making. Furthermore, the module detects underutilized critical information during the decision-making process and generates an information missing flag. Missing information flags alert the system to potential data loss or underutilization, providing a basis for data supplementation and optimization. Finally, the adaptive decision module encapsulates modal importance scores, data quality assessment results, and missing information flags into structured feedback information. This structured feedback facilitates system storage, management, and analysis, providing crucial support for system performance optimization and decision-making improvements.

[0067] It is understood that the adaptive decision module also needs to record the decision results output by the decision network, calculate the variance of the decision results based on Monte Carlo dropout sampling, generate a confidence score, associate the confidence score with the decision result, and record and associate explanation information with the decision result. The explanation information includes global explanations, modal explanations, and local explanations. The global explanation includes the task type, decision goal, and influencing factors; the modal explanation includes the modal importance score; and the local explanation includes the reasoning process for a specific decision point in the decision network. Finally, the adaptive decision module uses a pre-trained natural language generation model to convert the decision results, confidence scores, and explanation information into easy-to-understand textual explanations. The generated textual explanations are stored in the knowledge base. The knowledge base serves as the system's information storage center, storing a large amount of historical decision information and related explanations. These textual explanations provide an important reference for subsequent decision optimization. By analyzing and learning from the information in the knowledge base, the system can continuously improve its decision-making algorithms and models, enhancing its decision-making performance and intelligence.

[0068] 105. The meta-learning controller evaluates the system performance of the current multimodal data processing system and adjusts the system parameters based on the evaluation results. The system parameters include the hyperparameters and network structure of the specified modules. The specified modules include the multimodal encoding module, the cross-modal fusion module and the adaptive decision module.

[0069] In the embodiment of the present application, in order to effectively evaluate and optimize the performance of the multimodal data processing system, a key component, the meta-learning controller, is introduced, which plays a vital role in the system performance evaluation and parameter optimization process. Figure 5As shown, the meta-learning controller possesses powerful data processing and analysis capabilities. It accurately calculates multiple key performance indicators for the system in both historical and new tasks. Specifically, these indicators include average accuracy, inference latency, and energy efficiency. Average accuracy reflects the degree to which the system's output matches the actual result when completing a task and is an important indicator of the system's task execution accuracy. Inference latency reflects the time it takes for the system to convert input data into output, directly affecting the system's real-time responsiveness. Energy efficiency focuses on the system's energy consumption during task completion and is crucial for sustainable operation and cost control. By comprehensively considering these indicators, the meta-learning controller constructs a comprehensive performance score that comprehensively reflects the system's performance. This score intuitively demonstrates the system's overall performance in different task scenarios, providing an objective and accurate basis for subsequent system performance evaluation. During the evaluation process, the system's comprehensive performance score for the new task is compared against a preset threshold, a performance standard set based on the system's design goals and actual application requirements. If the evaluation results indicate that the system's overall performance score for the new task is at least as high as the preset threshold, this indicates that the current system parameter configuration can effectively meet the performance requirements of the new task and that the system's performance in terms of accuracy, inference latency, and energy efficiency meets the expected standards. In this case, to maintain system stability and reliability, the current system parameter configuration will be maintained to avoid unnecessary parameter adjustments that could adversely affect system performance. Conversely, if the evaluation results indicate that the system's overall performance score for the new task is below the preset threshold, this indicates that the current system parameter configuration may not effectively meet the requirements of the new task and that there is room for improvement in system performance. At this point, the system will initiate an optimization mechanism, retrieving historically optimal parameter configurations from a knowledge base for tasks similar to the current one. The knowledge base stores various parameter configurations accumulated from past tasks and their corresponding performance verification results. This historical data provides data support for system optimization. After obtaining the historically optimal parameter configurations for similar tasks, the system will adjust the current system parameters using a meta-learning algorithm. The meta-learning algorithm can quickly learn from the valuable information in historical data and optimize system parameters based on the characteristics and requirements of the current task, thereby improving the system's overall performance for the new task. Finally, to achieve continuous system evolution and performance improvement, the system stores optimized system parameters and corresponding verification results in a knowledge base. This new data further enriches the knowledge base, forming a continuously evolving configuration chain. As the system continuously processes new tasks, the data in the knowledge base is continuously updated and improved, providing a more accurate and comprehensive reference for future system optimization, enabling the multimodal data processing system to maintain excellent performance in a constantly changing task environment.

[0070] As you can understand, the knowledge base, as a crucial information repository for the system, contains an environmental dynamics model that is trained and optimized using extensive data. This model fully considers various environmental factors and their interrelationships, accurately capturing the dynamics of environmental states. By learning and analyzing historical environmental data, the environmental dynamics model simulates the temporal evolution of different environmental factors, providing a reliable basis for the system to predict future environmental state transitions. Therefore, to further enhance the system's performance and adaptability in complex and changing environments, during actual operation, the system proactively invokes the environmental dynamics model when it detects potential environmental changes or needs to address future task requirements. The system inputs current environmental state information into the environmental dynamics model and, leveraging its predictive capabilities, predicts environmental state transitions over a period of time. Based on these predictions, the system proactively optimizes the hyperparameters and network architecture of each module. Hyperparameters are crucial parameters that control the model's learning process, and different environmental conditions may require different hyperparameter settings for optimal model performance. Based on predicted environmental changes, the system dynamically adjusts the hyperparameters of each module to ensure the model can quickly and accurately learn and adapt to future environments. At the same time, the network structure will be optimized accordingly based on the prediction results. The design of the network structure directly affects the model's expressive power and computational efficiency. Different environmental conditions and task requirements may require different network structures to achieve optimal performance. The system will adjust and optimize the network structure of each module based on the characteristics and requirements of the future environment. For example, this may involve adding or reducing network layers and adjusting the number of neurons to improve the model's adaptability and processing capabilities in future environments. Through this mechanism of advance prediction and optimization, the system can prepare for environmental changes and adjust its parameters and structure in a timely manner, thereby maintaining good performance and stability in complex and changing environments.

[0071] The method provided in the embodiments of this application dynamically adjusts fusion weights through a multi-head cross-modal attention mechanism, enabling deep correlation and complementarity mining of multimodal information, improving the accuracy and robustness of multimodal understanding. Furthermore, through cross-task parameter optimization of the meta-learning controller, the system dynamically adapts to new scenarios and reduces performance fluctuations caused by environmental changes. Furthermore, through a closed-loop feedback mechanism, the data collection and processing processes are continuously optimized, enabling stable operation and continuous evolution of the system in complex scenarios.

[0072] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a multimodal data processing system, such as Figure 6 As shown, the system includes: an environment perception module 601, a multimodal encoding module 602, a cross-modal fusion module 603, an adaptive decision module 604, and a meta-learning controller 605.

[0073] The environment perception module 601 adjusts the environment perception strategy based on the feedback information transmitted by the adaptive decision module and collects multimodal data according to the environment perception strategy, wherein the feedback information includes but is not limited to the modality importance score, data quality assessment results and information missing flag;

[0074] The multimodal encoding module 602 encodes the multimodal data into a multimodal feature vector of the same dimension;

[0075] The cross-modal fusion module 603 uses a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature. The fusion weight is adjusted according to the task type of the current decision task and the input data characteristics;

[0076] The adaptive decision module 604 selects a decision network that matches the task type from a predefined network library according to the task type of the current decision task, inputs the fused features into the decision network, generates feedback information based on the decision process of the decision network, and transmits the feedback information to the environment perception module through a reverse communication channel;

[0077] The meta-learning controller 605 evaluates the system performance of the current multimodal data processing system and adjusts system parameters according to the evaluation results. The system parameters include hyperparameters and network structures of designated modules. The designated modules include a multimodal encoding module, a cross-modal fusion module, and an adaptive decision module.

[0078] In a specific application scenario, the system further includes: a knowledge base 606 .

[0079] The knowledge base 606 stores historical task model configurations of different task types, as well as decision processes, decision results, confidence scores and explanation information of different decision tasks, wherein the historical task model configurations include network structure, hyperparameter combination and pre-trained weight mapping relationship, and the explanation information includes global explanation, modal explanation and local explanation, wherein the global explanation includes task type, decision goal and influencing factors, the modal explanation includes modal importance score, and the local explanation includes the reasoning process of a specified decision point in the decision network.

[0080] The system provided in the embodiments of this application dynamically adjusts fusion weights through a multi-head cross-modal attention mechanism, enabling deep correlation and complementary mining of multimodal information, improving the accuracy and robustness of multimodal understanding. Furthermore, through cross-task parameter optimization of the meta-learning controller, the system dynamically adapts to new scenarios and reduces performance fluctuations caused by environmental changes. Furthermore, through a closed-loop feedback mechanism, the data collection and processing processes are continuously optimized, enabling stable operation and continuous evolution of the system in complex scenarios.

[0081] It should be noted that for other corresponding descriptions of the functional units involved in the multimodal data processing system provided in the embodiment of the present application, please refer to Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0082] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0083] like Figure 7 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a data relationship reconstruction method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a data relationship reconstruction method. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0084] The present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data relationship reconstruction method in any of the above embodiments.

[0085] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0086] The present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data relationship reconstruction method in any of the above embodiments.

[0087] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0088] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0089] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal data processing method, characterized in that: include: The environmental perception module adjusts the environmental perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environmental perception strategy, wherein the feedback information includes but is not limited to the modality importance score, the data quality assessment result and the information missing mark; The multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension; The cross-modal fusion module adopts a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature. The fusion weight is adjusted according to the task type of the current decision task and the input data characteristics; The adaptive decision module selects a decision network that matches the task type from a predefined network library according to the task type of the current decision task, inputs the fusion feature into the decision network, generates feedback information according to the decision process of the decision network, and transmits the feedback information to the environment perception module through a reverse communication channel; The meta-learning controller evaluates the system performance of the current multimodal data processing system and adjusts system parameters according to the evaluation results. The system parameters include hyperparameters and network structures of designated modules. The designated modules include the multimodal encoding module, the cross-modal fusion module and the adaptive decision module.

2. The method according to claim 1, characterized in that The environmental perception module adjusts the environmental perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environmental perception strategy, including: The environment perception module receives the feedback information, associates the feedback information with corresponding multimodal data according to the timestamp and task identifier carried in the feedback information, and stores the associated feedback information in a knowledge base; The environmental perception module adjusts the sampling parameters indicated by the data acquisition strategy according to the modal importance score in the feedback information, and adjusts the re-acquisition trigger threshold or the sampling parameters indicated by the data acquisition strategy according to the data quality assessment result in the feedback information, and adjusts the attention allocation result indicated by the attention allocation strategy according to the information missing flag in the feedback information, and adjusts the preprocessing rules indicated in the data preprocessing strategy according to the data quality assessment result in the feedback information, wherein the sampling parameters include but are not limited to sampling frequency and sampling accuracy, and the environmental perception strategy includes the data acquisition strategy, the attention allocation strategy, and the data preprocessing strategy; The environmental perception module obtains the multimodal data through multiple data acquisition sensors or API interfaces according to the sampling parameters indicated by the data acquisition strategy and the attention allocation results indicated by the attention allocation strategy, and preprocesses the multimodal data according to the preprocessing rules indicated by the data preprocessing strategy, and inputs the preprocessed multimodal data into the multimodal encoding module.

3. The method according to claim 1, characterized in that The multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension, including: The multimodal encoding module determines a feature extraction network corresponding to each modal data, wherein the text modal data uses a Transformer encoder, the image modal data uses a visual Transformer encoder or a CNN network, and the audio modal data uses a waveform encoder; The multimodal encoding module adjusts the encoding depth and encoding width corresponding to each feature extraction network according to the signal-to-noise ratio of the input data and the task complexity required by the task, and extracts data features corresponding to each modal data based on the adjusted feature extraction network; The multimodal encoding module maps the data features corresponding to each modality data to a shared semantic space based on contrastive learning technology to obtain a multimodal feature vector of the same dimension.

4. The method according to claim 1, wherein The cross-modal fusion module adopts a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature, including: The cross-modal fusion module uses a multi-head cross-modal attention mechanism to calculate the association weight between each modal feature vector to obtain an initial fusion weight, and constructs a task feature vector according to the task type, maps the task feature vector to a task-related weight through a deep neural network, and determines the quality score and complementarity score corresponding to the multimodal feature vector, and generates a data feature-related weight based on the quality score and the complementarity score, wherein the quality score is calculated based on the signal-to-noise ratio and completeness of each modal feature vector, and the complementarity score is calculated based on the mutual information between the feature vectors of different modalities. The task type is obtained by identifying the user instruction; The cross-modal fusion module performs a weighted summation on the initial fusion weight, the task-related weight, and the data feature-related weight based on the balance coefficient to obtain a comprehensive weight, constructs a task loss function according to the task performance indicator, continuously optimizes the balance coefficient using the task loss function and a weight regularization term through a backpropagation algorithm to obtain an optimized comprehensive weight, and normalizes the optimized comprehensive weight using a normalization function to obtain the fusion weight; The cross-modal fusion module performs weighted fusion on the multimodal feature vectors based on the fusion weights to obtain the fusion features.

5. The method according to claim 1, wherein The adaptive decision module selects a decision network that matches the task type from a predefined network library according to the task type of the current decision task, inputs the fusion feature into the decision network, and generates feedback information according to the decision process of the decision network, including: The adaptive decision module parses the user instruction and determines the intention encoding vector of the user instruction through a pre-trained language model, performs similarity matching between the intention encoding vector and the task feature vector in the preset task library, determines the task type based on the matching result, and loads the adapted decision network architecture from the predefined network library based on the task type; The adaptive decision module inputs the fusion features into the decision network, records the decision process, and extracts the attention weights of different modal feature channels in the decision network based on the decision process to construct a modal contribution matrix to obtain the modal importance score, and determines the signal-to-noise ratio and integrity of the input data based on the decision process, generates the data quality assessment result according to the signal-to-noise ratio and the integrity, and detects underused key information based on the decision process to generate the information missing identifier, and encapsulates the modal importance score, the data quality assessment result and the information missing identifier into structured feedback information, wherein the modal importance score is used to indicate the contribution ratio of each modal data to the decision result.

6. The method according to claim 5, characterized in that The method further comprises: The adaptive decision module records the decision results output by the decision network, calculates the variance of the decision results based on Monte Carlo Dropout sampling, generates a confidence score, associates the confidence score with the decision results, and records explanation information, associates the explanation information with the decision results, the explanation information includes a global explanation, a modal explanation, and a local explanation, wherein the global explanation includes a task type, a decision goal, and an influencing factor, the modal explanation includes a modal importance score, and the local explanation includes a reasoning process of a specified decision point in the decision network; The adaptive decision module converts the decision result, the confidence score and the explanation information into a textual explanation using a pre-trained natural language generation model, and stores the textual explanation in a knowledge base; The meta-learning controller evaluates the system performance of the current multimodal data processing system, queries the historical task model configuration in the knowledge base according to the evaluation result, and adjusts the current system parameters according to the historical task model configuration.

7. A multimodal data processing system, characterized in that: The system includes: an environment perception module, a multimodal encoding module, a cross-modal fusion module, an adaptive decision module and a meta-learning controller; The environmental perception module adjusts the environmental perception strategy according to the feedback information transmitted by the adaptive decision module, and collects multimodal data according to the environmental perception strategy, wherein the feedback information includes but is not limited to the modality importance score, the data quality assessment result and the information missing mark; The multimodal encoding module encodes the multimodal data into a multimodal feature vector of the same dimension; The cross-modal fusion module uses a multi-head cross-modal attention mechanism to calculate the fusion weight of each modal data, and fuses the multi-modal feature vector according to the fusion weight corresponding to each modal data to obtain the fusion feature. The fusion weight is adjusted according to the task type of the current decision task and the input data characteristics; The adaptive decision module selects a decision network that matches the task type from a predefined network library according to the task type of the current decision task, inputs the fusion feature into the decision network, generates feedback information according to the decision process of the decision network, and transmits the feedback information to the environment perception module through a reverse communication channel; The meta-learning controller evaluates the system performance of the current multimodal data processing system and adjusts system parameters based on the evaluation results. The system parameters include hyperparameters and network structures of designated modules. The designated modules include the multimodal encoding module, the cross-modal fusion module, and the adaptive decision module.

8. The multimodal data processing system according to claim 7, wherein: The system further includes: a knowledge base; The knowledge base stores historical task model configurations of different task types, as well as decision processes, decision results, confidence scores and explanation information of different decision tasks, wherein the historical task model configurations include network structure, hyperparameter combination and pre-trained weight mapping relationship, and the explanation information includes global explanation, modal explanation and local explanation, wherein the global explanation includes task type, decision goal and influencing factors, the modal explanation includes modal importance score, and the local explanation includes the reasoning process of specified decision points in the decision network.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images