Decision-making method and device guided by multi-modal semantic map, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as financial science and technology and medical health, and discloses a decision-making method and device guided by a multi-modal semantic map, equipment and a medium. Extracting a visual feature vector, a language feature vector and an action feature vector, splicing to generate a multi-modal initial feature, mapping the multi-modal initial feature to a shared semantic space, constructing a multi-modal semantic map, and inputting a map-guided attention mechanism to generate a cross-modal alignment feature; the cross-modal alignment features and task targets are input into a meta-learner to generate task adaptability features, the task adaptability features are input into a parallel reasoning network to execute subtasks in parallel, and a gating fusion network integrates output results to generate a global decision. According to the method, cross-modal semantic association and task adaptability are enhanced through the combination of shared semantic space mapping, map guiding attention and a meta learning device, and the accuracy and efficiency of multi-modal decision making are improved through the combination of parallel reasoning and gating fusion.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a decision-making method, apparatus, device, and storage medium guided by multimodal semantic graphs. Background Technology

[0002] Current Vision-Language-Action (VLA) models generally suffer from insufficient semantic correlation mining when processing multimodal data. Existing technologies often model visual, linguistic, and action modalities independently, integrating them only through feature concatenation or shallow weighted fusion. This makes it difficult to capture the complex semantic relationships between modalities at a fine-grained level, especially limiting tasks requiring a deep understanding of the correspondence between visual scenes, linguistic descriptions, and action execution. In the fintech sector, this deficiency prevents models from establishing accurate semantic alignment when processing financial product features described in user language, financial contract text within visual data, and user interaction data, impacting the accuracy and efficiency of intelligent customer service. In the healthcare sector, models struggle to organically integrate patient language expressions, imaging data, and physician workflows, failing to effectively support comprehensive diagnostic or health management services based on multimodal medical data.

[0003] Furthermore, existing technologies generally employ serial inference mechanisms, resulting in low computational efficiency and making it difficult to meet the demands of real-time multimodal tasks requiring rapid response. In the fintech sector, facing scenarios with high concurrency queries from multiple users and dynamic interactions, existing models cannot process financial knowledge graphs, text semantics, and user action input in parallel, limiting the real-time decision-making capabilities of intelligent services in complex business processes. In the healthcare sector, existing models, when processing multi-source medical data, involve lengthy inference processes and consume significant computational resources, making it difficult to achieve rapid response and real-time decision support in clinical scenarios.

[0004] Current technologies generally lack the ability to accurately focus on key information in multimodal data. In complex environments containing a large amount of redundant or non-critical information, this can easily lead to biases or errors in inference results. In the fintech sector, this makes it difficult for models to accurately extract core elements related to financial risk assessment, thus affecting the accuracy of risk control systems. In the healthcare sector, the lack of accurate capture of key diagnostic features can easily increase the risk of misdiagnosis or missed diagnosis. Current models also lack the ability to process low-frequency but crucial long-tail semantic information, severely limiting their generalization and adaptability when faced with complex and diverse business needs. Summary of the Invention

[0005] The main objective of this invention is to provide a decision-making method, apparatus, device, and storage medium guided by multimodal semantic graphs, aiming to solve the technical problem that existing technologies struggle to achieve deep cross-modal semantic association and alignment when processing visual, linguistic, and action multimodal data, resulting in insufficient accuracy and adaptability in complex tasks.

[0006] To achieve the above objectives, the present invention provides a decision-making method guided by multimodal semantic graphs, comprising:

[0007] Acquire visual data, text data, and action data; extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively; concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features;

[0008] The visual feature vector, language feature vector, and action feature vector are respectively mapped to a shared semantic space to generate shared semantic space features;

[0009] A multimodal semantic graph is constructed based on the aforementioned multimodal initial features;

[0010] The initial multimodal features and the multimodal semantic graph are input into a graph-guided attention mechanism to generate cross-modal aligned features;

[0011] The cross-modal alignment features and the task objective are input into the meta-learner to generate task-adaptive features;

[0012] The task-adaptive features are input into a parallel inference network, in which visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks, and cross-modal integrated reasoning sub-tasks are executed in parallel to process the feature content corresponding to each sub-task in the task-adaptive features and generate the output results of each sub-task.

[0013] The output results of each subtask are integrated through a gated fusion network to generate a global decision.

[0014] Furthermore, to achieve the above objectives, the present invention provides a decision-making device guided by a multimodal semantic graph, comprising:

[0015] The multimodal feature extraction module is used to acquire visual data, text data, and action data, extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively, and concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features;

[0016] The semantic mapping module is used to map the visual feature vector, language feature vector and action feature vector to the shared semantic space respectively, and generate shared semantic space features;

[0017] A semantic graph construction module is used to construct a multimodal semantic graph based on the multimodal initial features;

[0018] The graph-guided attention module is used to input the multimodal initial features and the multimodal semantic graph into the graph-guided attention mechanism to generate cross-modal aligned features;

[0019] The task adaptation module is used to combine the cross-modal alignment features with the task target input meta-learner to generate task-adaptive features;

[0020] The parallel reasoning module is used to input the task-adaptive features into the parallel reasoning network, and execute visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks and cross-modal integrated reasoning sub-tasks in parallel in the parallel reasoning network, so as to process the feature content corresponding to each sub-task in the task-adaptive features and generate the output results of each sub-task.

[0021] The decision fusion module is used to integrate the output results of each subtask through a gated fusion network to generate a global decision.

[0022] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multimodal semantic graph-guided decision-making program stored in the memory and executable on the processor, wherein when the multimodal semantic graph-guided decision-making program is executed by the processor, it implements the steps of the multimodal semantic graph-guided decision-making method as described above.

[0023] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multimodal semantic graph-guided decision-making program, wherein when the multimodal semantic graph-guided decision-making program is executed by a processor, it implements the steps of the multimodal semantic graph-guided decision-making method as described above.

[0024] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a multimodal semantic graph-guided decision-making method, apparatus, device, and medium, comprising: acquiring visual data, text data, and action data; extracting visual feature vectors, language feature vectors, and action feature vectors respectively; concatenating them to generate multimodal initial features; mapping the visual feature vectors, language feature vectors, and action feature vectors to a shared semantic space to generate shared semantic space features; constructing a multimodal semantic graph based on the multimodal initial features; generating cross-modal alignment features using a graph-guided attention mechanism; generating task-adaptive features by combining the cross-modal alignment features with the task objective input meta-learner; inputting the task-adaptive features into a parallel inference network; executing visual inference sub-tasks, language inference sub-tasks, action inference sub-tasks, and cross-modal comprehensive inference sub-tasks; generating the output results of each sub-task; and integrating the output results of each sub-task through a gating fusion network to generate a global decision. This invention fully explores and aligns deep semantic relationships between different modalities by using a shared semantic space, a graph-guided attention mechanism, and a meta-learner. It also achieves efficient and accurate decision-making in multimodal tasks through parallel inference and a gated fusion network. This invention can solve the problems of insufficient cross-modal semantic relationships and poor adaptability in traditional models, and effectively improve the accuracy and real-time performance in complex multimodal task scenarios. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0026] Figure 1 This is a schematic diagram of an application environment for a multimodal semantic graph-guided decision-making method according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart illustrating an embodiment of the multimodal semantic graph-guided decision-making method of the present invention;

[0028] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multimodal semantic graph-guided decision-making device of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] The multimodal semantic graph-guided decision-making method provided in this embodiment of the invention can be applied to, for example, Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can obtain visual, text, and action data from the user terminal, extract visual, linguistic, and action feature vectors respectively, concatenate them to generate multimodal initial features, map the visual, linguistic, and action feature vectors to a shared semantic space to generate shared semantic space features, construct a multimodal semantic graph based on the multimodal initial features, generate cross-modal alignment features using an attention mechanism guided by the input graph, input the cross-modal alignment features and the task objective into a meta-learner to generate task-adaptive features, input the task-adaptive features into a parallel inference network, execute visual inference sub-tasks, linguistic inference sub-tasks, action inference sub-tasks, and cross-modal integrated inference sub-tasks, generate the output results of each sub-task, integrate the output results of each sub-task through a gating fusion network to generate a global decision. This invention, through a shared semantic space, a graph-guided attention mechanism, and a meta-learner, fully mines and aligns deep semantic relationships between different modalities. It achieves efficient and accurate decision-making in multimodal tasks through parallel inference and a gated fusion network, addressing the problems of insufficient cross-modal semantic relationships and poor adaptability in traditional models. This effectively improves accuracy and real-time performance in complex multimodal task scenarios. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0033] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multimodal semantic graph-guided decision-making method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the multimodal semantic graph-guided decision-making method proposed in this invention includes the following steps:

[0035] S10, acquire visual data, text data and action data, extract visual feature vectors, language feature vectors and action feature vectors from the visual data, text data and action data respectively, and concatenate the visual feature vectors, language feature vectors and action feature vectors to generate multimodal initial features;

[0036] In this embodiment, when acquiring visual data, text data, and motion data, visual data can be obtained by acquiring single-frame images or continuous video frames through image acquisition devices, including cameras, webcams, and other devices that support visible light or infrared imaging. The source of visual data can be on-site acquisition or a historical image database. Text data refers to language description information represented by character sequences, which can originate from user-input natural language, preset instructions, or text content in a database. Motion data is the position, velocity, and acceleration data of a human or robot in a time-series dimension acquired through motion sensors. Motion sensors include inertial measurement units, gyroscopes, accelerometers, and other devices used to acquire multi-dimensional temporal motion trajectories.

[0037] When extracting visual feature vectors from visual data, deep neural networks are first used for multi-layer feature representation, extracting global contextual information and local detail features layer by layer. For example, convolutional neural networks are used to convolve images layer by layer and perform pooling operations, outputting high-dimensional feature vectors with spatial and semantic hierarchical information. Visual feature vectors originate from the results of multi-channel processing of visual data and can include attributes such as object boundaries, texture patterns, and color distribution. When extracting linguistic feature vectors from text data, word segmentation and part-of-speech tagging algorithms are used to decompose the text into basic word units and label grammatical categories. Linguistic feature vectors are mapped to a high-dimensional space using word embedding techniques, which can express the semantic similarity and contextual relationships of words. For example, a bidirectional encoder representation model is used to process text and output sentence-level semantic vectors. When extracting action feature vectors from action data, temporal analysis models such as long short-term memory networks are used to model the sequence data collected by motion sensors, extracting the time dependencies of multi-dimensional action state changes. Action feature vectors include attributes such as the amplitude, frequency, and direction of the action, used to describe the dynamic process.

[0038] The operation of concatenating visual, linguistic, and action feature vectors essentially involves joining these three different sources of feature vectors dimensionally to form a unified representation. During concatenation, the integrity of each feature vector within its original space is maintained. By aligning the order of the feature dimensions, the concatenated result is ensured to be accurately decoded in subsequent processing. The concatenated vector structure preserves the independence of visual, linguistic, and action information, while providing rich semantic support for subsequent multimodal processing through high-dimensional representation.

[0039] Generating multimodal initial features refers to using the concatenated vectors as the overall input, serving as a unified representation of visual, linguistic, and action data. This set can be directly used as the input basis for multimodal information analysis. Multimodal initial features contain structural, temporal, and semantic information from different modalities, forming a comparable and computable multidimensional representation within a shared space.

[0040] Visual data acquisition can be achieved by using industrial cameras to capture real-time images of workpieces on the production line and inputting them into an image processing module, or by acquiring patient medical images from hospital imaging equipment. Text data acquisition can be achieved by receiving user speech-to-text via an interface or extracting transaction description text from a financial business platform. Motion data acquisition can utilize sensor arrays from robot end effectors or wearable motion tracking devices for patients. Visual feature vector extraction can employ ResNet or EfficientNet networks and fine-tune them using pre-trained models to adapt to the scene data. Language feature vector extraction can combine word embedding models from natural language processing with context sequence modeling algorithms, such as using BERT or RoBERTa models to output context embedding representations. Motion feature vector extraction can use BiLSTM networks to encode multidimensional acceleration and angular velocity sequences. During concatenation, matrix concatenation operations are used to connect the visual, language, and motion feature matrices along the column direction, ensuring that the order of feature dimensions after concatenation is consistent with the input. The network depth, output dimensions, and concatenation order for extracting each modality can be adjusted for different application scenarios to adapt to varying requirements of visual resolution, language complexity, and motion sampling frequency.

[0041] Example: In the field of healthcare, medical robots can use cameras to capture local images of patients as visual data, use voice descriptions from medical staff to record operational instructions as text data, and use built-in sensors to collect the movement trajectory of the robotic arm as motion data. The three data are then combined to form multimodal initial features, providing input for precise assisted operations.

[0042] In the fintech business, customers upload product photos as visual data via mobile devices, input purchase requests as text data, and record user interaction actions as action data. These three types of data are then combined to form multimodal initial features, which provide input for intelligent recommendation algorithms to achieve personalized recommendations.

[0043] This embodiment can simultaneously collect and process multimodal data through the above method, and obtain a unified feature representation that integrates visual information, language semantics and action dynamics. In multimodal tasks, it avoids isolated processing of modalities, enhances the consistency of expression and information complementarity between different modalities, and provides high-quality input for subsequent cross-modal semantic association, thereby improving the semantic understanding and task adaptation efficiency of multimodal tasks.

[0044] S20, the visual feature vector, language feature vector and action feature vector are respectively mapped to the shared semantic space to generate shared semantic space features;

[0045] In this embodiment, visual feature vectors, linguistic feature vectors, and action feature vectors are mapped to a shared semantic space. This means transforming multimodal representations with significant differences in source and data structure into a unified representation with the same distribution characteristics, comparability, and interactivity. Visual feature vectors are high-dimensional vectors extracted from image or video frames, containing spatial information such as texture, color, and shape. Linguistic feature vectors are high-dimensionally embedded using a language encoder, incorporating text segmentation, part-of-speech tagging, and contextual relationships, and include syntactic and semantic attributes. Action feature vectors encode motion sensor data through temporal modeling, containing dynamic processes and motion trajectory features. These vectors are heterogeneous in their original space, and direct comparison would be incomparable due to differences in feature distribution.

[0046] To eliminate modal differences, visual semantic mapping networks, linguistic semantic mapping networks, and action semantic mapping networks are needed to transform their respective feature vectors. Visual semantic mapping networks can employ a multilayer perceptron structure, extracting common representations of visual semantic expression through layer-by-layer linear transformations and non-linear activations of visual feature vectors. Linguistic semantic mapping networks can also use a multilayer perceptron structure to transform linguistic feature vectors into embedding representations comparable to those in the visual space. Action semantic mapping networks can use a time-series coding network followed by a multilayer perceptron to transform temporal dynamic features into static spatial embedding vectors.

[0047] The essence of the mapping operation is to force the embedding results of different modalities to satisfy a uniform distribution constraint in the semantic space by sharing a parameter space or a target alignment loss function. The shared semantic space refers to a high-dimensional continuous space where the distances within the space reflect the proximity of semantics across different modalities, facilitating subsequent calculations of multimodal semantic associations. Generating shared semantic space features involves combining the mapped visual, linguistic, and action embedding vectors into a unified semantic alignment representation, which serves as input for subsequent processing.

[0048] The visual semantic mapping network can be configured with a three-layer fully connected structure, each layer followed by a non-linear activation function such as ReLU or GELU, and batch normalization is applied to the output of each layer to accelerate convergence. The language semantic mapping network can have a similar network structure, maintaining the same output dimension as the visual semantic mapping network to ensure alignment of their outputs in a shared space. The action semantic mapping network can use a temporal convolutional network to process action sequences, then feed the output as input to a multilayer perceptron, adjusting the output dimension to match both visual and language dimensions. When training the three networks separately, a multimodal contrastive learning loss function can be used to compress samples from different modalities but corresponding to the same semantics within a shared semantic space, while dispersing samples from different semantics within the space. The number of network layers, hidden units, activation function type, and regularization strategy can be adjusted according to application requirements to adapt to different data scales and distribution characteristics.

[0049] Example: In the healthcare field, the visual features generated from a patient's surgical video frame, along with the doctor's voice instructions and the trajectory of surgical instruments, are mapped to a shared semantic space through a visual semantic mapping network, a language semantic mapping network, and an action semantic mapping network, respectively. This ensures that information from different sources is expressed consistently in the shared space, enabling the auxiliary decision-making algorithm to perform unified analysis.

[0050] In the fintech business, user-uploaded product images, shopping needs descriptions, and browsing behavior paths are transformed into comparable embedding representations in a shared semantic space after passing through a mapping network. This enables the recommendation system to accurately identify user needs across modalities and provide highly personalized product recommendations.

[0051] This embodiment, by mapping to a shared semantic space, enables the visual, linguistic, and action vectors to be expressed as consistent and comparable vectors in a unified space, even when the original modal distributions are inconsistent. This eliminates the semantic gap between modalities, enhances the collaborative expression capability of multimodal features, provides standardized input for subsequent cross-modal semantic association and task adaptation, and strengthens the robustness and generalization performance of the overall system under multimodal tasks.

[0052] S30, Construct a multimodal semantic graph based on the initial multimodal features;

[0053] In this embodiment, a multimodal semantic graph is constructed based on multimodal initial features. This involves analyzing the semantic elements and their relationships contained within the multimodal initial features obtained by concatenating visual feature vectors, linguistic feature vectors, and action feature vectors, and organizing these elements and relationships into a graph structure representation. The multimodal initial features, as the result of multimodal data fusion, include spatial semantic information from images, syntactic semantic information from linguistic text, and dynamic behavioral semantic information from action sequences. By analyzing these features, key elements reflecting semantic content are identified, such as objects, actions, and expected results, and further categorized into object nodes, action nodes, and result nodes. Each type of node represents its features in vector form; for example, object nodes correspond to visual semantic features, action nodes correspond to action semantic features, and result nodes correspond to linguistic descriptive semantic features.

[0054] Edges between nodes represent relationships between different types of semantic elements. For example, a tool relationship edge can be established between an object node and an action node, indicating that a specific object is a tool for performing a specific action. A causal relationship edge can be established between an action node and a result node, indicating that a specific action leads to a certain result. The message passing mechanism of graph neural networks can be applied to pass and aggregate features of neighboring nodes between nodes, achieving contextual enhancement of node features. After message passing and node feature updates, a multimodal semantic graph containing nodes, relationship edges, and their feature representations is finally obtained.

[0055] Multimodal semantic graph construction can be implemented as follows: First, object candidate regions related to visual content are separated from the initial multimodal features. The category of each object is determined using an object detection model and designated as an object node. Second, phrase and sentence structure are analyzed using linguistic features to extract action keywords from the instructions, which are then used as action nodes. Then, result nodes are defined by combining action features with the operation results described in the language. Each node is accompanied by an embedding vector of its corresponding modality as a feature. Tool-related edges can be determined using linguistic dependencies and contextual semantic similarity measures; for example, establishing a tool relationship between "pen" and "write" in "write with a pen." Causal-related edges can be determined by the degree of contextual coupling between the action and the result in the semantic graph. A graph neural network is used for message passing layer by layer, aggregating the feature vectors of adjacent nodes at each layer to update the node representation. Global association information of nodes can be enhanced through multi-layer graph convolution operations. The final output graph structure can be stored in an adjacency matrix and a node feature matrix for subsequent computation.

[0056] Example: In the healthcare field, by analyzing patient images, doctors' operational language, and surgical operation trajectories, a multimodal semantic graph is constructed, consisting of object nodes such as "scalpel," action nodes such as "resection," and result nodes such as "complete tissue resection." Tool relationship edges mark "scalpel" as the tool used for "resection," while causal relationship edges indicate that "resection" leads to "complete tissue resection," providing support for surgical procedure analysis and guidance.

[0057] In the fintech business, for the insurance claims process, the accident photos, voice descriptions and click behaviors uploaded by users are parsed into multimodal initial features, which are organized into object nodes such as "vehicle" and "damaged", action nodes such as "reporting the incident" and "claiming", and result nodes such as "successful payment" through a graph, to build a clear multimodal semantic graph, supporting semantic parsing and cross-modal verification of multimodal data in complex claims processes.

[0058] This embodiment constructs a multimodal semantic graph, organizing visual, linguistic, and action features into a graph representation with a clear structure. This enhances the organization and parsing capabilities of multimodal information, transforming originally heterogeneous and disordered multimodal data into a unified representation format that can be used for subsequent graph computation and association reasoning. This strengthens the depth of semantic understanding and lays the data foundation for subsequent graph-guided cross-modal association modeling.

[0059] S40, input the multimodal initial features and the multimodal semantic graph into the graph-guided attention mechanism to generate cross-modal aligned features;

[0060] In this embodiment, a multimodal initial feature and a multimodal semantic graph-guided attention mechanism are used to perform fine-grained alignment of semantic relationships between different modalities. In this process, the multimodal initial feature represents a set of feature vectors fused from visual, linguistic, and action information, containing multidimensional feature representations of object appearance, linguistic description, and action trajectory. The multimodal semantic graph provides structured associations between multimodal semantic elements, with nodes representing key semantic elements and edges representing semantic relationships between elements. The graph-guided attention mechanism uses the node features of the multimodal semantic graph as contextual information to guide the calculation of the association strength between each feature element in the multimodal initial feature and the graph nodes. During the calculation, the similarity scores between the feature element vector and the node feature vector are non-linearly transformed using an activation function to form association weight values. These weight values ​​are then normalized to form a standardized attention weight distribution.

[0061] By utilizing a standardized attention weight distribution, different elements in the initial multimodal features are weighted and aggregated according to their semantic relevance to the graph nodes. This weighted aggregation process not only strengthens the feature elements that are semantically strongly related to the graph, but also weakens noise and irrelevant information, thereby achieving precise semantic-level alignment between visual, linguistic, and action modalities. Finally, cross-modal aligned features are generated as input for subsequent tasks.

[0062] This mechanism can be implemented as follows: First, a linear projection is calculated for each feature element in the initial multimodal features and each node in the multimodal semantic graph. A similarity score is formed through inner product or a learnable parameter matrix mapping, and a non-linear adjustment is achieved using an activation function (such as LeakyReLU) to enhance discriminability. Then, Softmax normalization is applied to the similarity scores of all nodes to obtain the standardized attention weights for each node relative to the current feature element. For a feature element, the standardized attention weights are weighted and summed with the feature representations of the multimodal semantic graph nodes to form a weighted graph feature. This process is repeated for all elements in the initial multimodal features to obtain a weighted aggregated feature set guided by the semantic graph, i.e., cross-modal aligned features. This process can batch process multimodal features in a parallel computing architecture, improving processing efficiency. The normalized distribution of the weights can dynamically change at the granularity of each feature element to adapt to different contexts.

[0063] Example: In the healthcare field, the process of generating cross-modal aligned features can be used in surgical navigation. By performing graph-guided attention-weighted aggregation on surgical scene images, voice commands, and surgical action data, feature representations that are strictly consistent with the semantics of surgical steps and surgical tools are obtained, thereby improving the accuracy of the surgical assistance system in understanding multimodal inputs.

[0064] In the fintech business, user-uploaded photos, descriptions, and actions in claims scenarios are integrated into multimodal initial features. Through an insurance business knowledge graph-guided attention mechanism, multimodal semantic information related to claims is aligned to form complete and accurate cross-modal aligned features, providing high-quality data support for automated claims decision-making.

[0065] This embodiment uses a multimodal semantic graph as contextual guidance. The attention mechanism can dynamically allocate weights based on the cross-modal relationships described in the graph, effectively eliminating the problem of insufficient shallow semantic coupling caused by simple splicing in traditional models, enhancing the deep semantic relationships between different modal features, and finally generating cross-modal aligned features that can more accurately express the unified semantics of multimodal data, laying a solid foundation for subsequent multi-task processing.

[0066] S50, the cross-modal alignment features are input into the task target meta-learner to generate task-adaptive features;

[0067] In this embodiment, the cross-modal alignment feature comprises a multi-dimensional vector representation that has achieved semantic consistency across visual, linguistic, and action modalities through semantic graph guidance. The task objective is a feature-based expression of task requirements, which can be a text description, category label, or other structured representation. Both are input into the meta-learner, which uses a few-shot fast adaptation mechanism to extract general knowledge across different tasks and sensitively capture the specific requirements of the current task. During the input process, the task objective is first encoded into a task objective vector with the same dimension as the cross-modal alignment feature, enabling the task objective to be computed and interact with the feature in a unified space. Subsequently, the meta-learner receives the cross-modal alignment feature and the task objective vector as joint input, extracts the general knowledge representation shared between different tasks, and simultaneously identifies the difference features between the current task and the training history tasks.

[0068] The meta-learner, through its parameterized model structure, rapidly generates transferable model parameters for the current task upon input. These parameters are used to adjust a feature transformation matrix, which remaps cross-modal aligned features to a feature space best suited for the current task. The adjusted feature transformation matrix processes the cross-modal aligned features, ultimately outputting task-adaptive features. These task-adaptive features are specific and transferable, enhancing the accuracy and adaptability of subsequent multi-task parallel inference networks.

[0069] This process can be implemented as follows: First, the task objective is encoded into a task objective vector using a multilayer perceptron or recurrent neural network. This vector, along with the cross-modal alignment features, serves as the input to the meta-learner. The meta-learner can be based on a model-independent meta-learning algorithm (MAML). During the training phase, its initial parameter matrix is ​​updated using different task samples. During the application phase, a few gradient updates are performed quickly for the current task, generating a transferable model parameter matrix. This parameter matrix is ​​then used to adjust the weights of a set of learnable feature transformation matrices. After processing with the adjusted matrix, the cross-modal alignment features fully absorb the knowledge of the current task and suppress irrelevant information, forming task-adaptive feature outputs. This process can be batch-parallel computed in a GPU-accelerated architecture, adapting to changes in different task objective dimensions and cross-modal features.

[0070] Example description: In the field of healthcare, this can be used in personalized health consultation. Cross-modal alignment features are jointly input into a meta-learner along with the patient's current health consultation task objective to generate health risk task adaptive features that meet individual needs, supporting subsequent intelligent health recommendations.

[0071] In the fintech business, intelligent financial customer service assistants can jointly input cross-modal alignment features of customer-uploaded invoice images, text inquiries, and operational behaviors with the current customer service task objectives into a meta-learner to quickly generate task-adaptive features. This enables the intelligent assistant to quickly adapt to different customer contexts and output more accurate service suggestions.

[0072] This embodiment inputs cross-modal alignment features and task objectives into a meta-learner and leverages the meta-learner's rapid adaptability to dynamically adjust feature mappings. This enables the model to quickly adapt to current task requirements with minimal data, achieving efficient transfer across domains and tasks. This approach effectively overcomes the limitation of traditional models requiring full retraining, significantly improving the applicability and response efficiency of multimodal models in dynamic task environments.

[0073] S60, the task adaptive features are input into the parallel inference network, and the visual reasoning sub-task, language reasoning sub-task, action reasoning sub-task and cross-modal integrated reasoning sub-task are executed in parallel in the parallel inference network to process the feature content corresponding to each sub-task in the task adaptive features and generate the output results of each sub-task.

[0074] In this embodiment, task-adaptive features, as input data, have been fully integrated with visual, linguistic, and action multimodal information through previous steps and optimized for the current task. After being input into the parallel inference network, the first step is to allocate the dimensions related to visual objects, semantic content, action trajectories, and multimodal decision-making within the task-adaptive features to the corresponding inference paths according to predefined feature mapping rules. This feature allocation operation ensures that the different modality-related content contained in the task-adaptive features can be efficiently assigned to visual inference subtasks, linguistic inference subtasks, action inference subtasks, and cross-modal integrated inference subtasks.

[0075] The visual reasoning subtask utilizes a visual semantic analysis module to specifically handle dimensions related to visual objects, such as detecting key objects in images and identifying visual scene attributes. The language reasoning subtask processes semantic content-related dimensions using a semantic parsing model, with tasks including entity extraction, relation recognition, and contextual understanding. The action reasoning subtask employs a motion trajectory analysis module to parse the feature content of the action trajectory dimension, such as action sequence modeling and motion intent prediction. The cross-modal integrated reasoning subtask processes features across the cross-modal decision dimension through a multimodal relation modeling module, integrating the semantic connections between visual, linguistic, and action reasoning results to generate a comprehensive reasoning output.

[0076] The four subtasks are executed in parallel within a unified hardware resource environment during computation. The consistency and timing coordination of the results of the four subtasks are ensured through the synchronization scheduling mechanism of the parallel inference network. Finally, the independent inference result of each subtask is output, which serves as the input to the subsequent global decision module.

[0077] This can be implemented as follows: Task-adaptive features are divided into four sub-task input channels by a feature allocation unit according to dimensional label or semantic label mapping rules. The visual reasoning sub-task uses a convolutional neural network combined with an attention mechanism to extract image features; the language reasoning sub-task uses a pre-trained language model combined with a context window to process the input language features; the action reasoning sub-task uses a temporal modeling network, such as a long short-term memory network, to process action trajectory features; and the cross-modal integrated reasoning sub-task interacts with the aforementioned three types of features through a multimodal tensor fusion module. The entire process is synchronously scheduled in a GPU or high-performance parallel computing environment, and the execution of inference computation is accelerated through pipelined parallelization and batch processing, supporting real-time processing of high-concurrency input tasks.

[0078] Example description: In the field of healthcare, this can be used in intelligent diagnostic robots. When the input includes patient medical record text, image data, and body posture data, the visual reasoning subtask can analyze abnormal areas in medical images, the language reasoning subtask can analyze the key points of the condition described in the medical record, the action reasoning subtask can assess motion abnormalities in body posture data, and the cross-modal integrated reasoning subtask integrates the results of the three to form a preliminary diagnostic suggestion.

[0079] In the fintech business, intelligent financial customer service can use customer behavior logs, verbal inquiries, and uploaded invoice images as task-adaptive feature inputs. The visual reasoning subtask identifies the invoice content, the verbal reasoning subtask parses the user's intent, the action reasoning subtask analyzes the operation trajectory, and the cross-modal integrated reasoning subtask integrates these contents to quickly generate targeted service response suggestions.

[0080] This embodiment significantly improves the inference efficiency of multimodal tasks and reduces processing latency by inputting task-adaptive features into a parallel inference network and performing feature content partitioning and parallel computation on four sub-tasks. Each sub-task performs specialized optimization processing on specific modality-related content, improving the processing accuracy of vision, language, action, and multimodal decision-making, effectively meeting the accuracy and efficiency requirements of complex multimodal tasks in real-time interactive scenarios.

[0081] The S70 integrates the output results of each subtask through a gated fusion network to generate a global decision.

[0082] In this embodiment, the output results of each subtask include the visual reasoning result of the visual reasoning subtask, the language reasoning result of the language reasoning subtask, the action reasoning result of the action reasoning subtask, and the cross-modal integrated reasoning result of the cross-modal integrated reasoning subtask. These results are collected as inputs and fed into a gated fusion network. The gated fusion network has dynamic weight adjustment capabilities. Through its internal gating units, it can calculate dynamic fusion weights based on the feature distribution of different task output results and the contextual relevance of the current task objective. Each subtask output result is first mapped to a unified dimension in the gated fusion network through a separate linear transformation unit, ensuring that the results of different subtasks reside in the same feature space. The gating unit further applies a Sigmoid function or other nonlinear activation to the mapped results, calculates dynamic weight coefficients, and adjusts the contribution of each subtask result. The dynamic weights are multiplied one by one with the unified-dimensional subtask results, and then fusion is completed through weighted summation. The fusion result serves as the global decision output, used to comprehensively express the global understanding and task response of multimodal information.

[0083] The gated fusion network can be designed as a parallel structure, where the four types of results—visual, language, action, and cross-modal—each input independent weight calculation channels. These weight calculation channels use trainable gating function modules, such as a multilayer perceptron combined with a sigmoid nonlinear activation function, to dynamically estimate the contribution of each type of result. The weight estimates are multiplied by the corresponding subtask results and then aggregated into a single global output feature vector through a weighted accumulator layer. Residual connections can be further used to directly superimpose some key subtask results into the global output, enhancing the preservation of important information. In hardware deployment, parallel matrix operations can be used to achieve efficient execution of weight estimation, weighted calculation, and output feature synthesis to support real-time interactive tasks.

[0084] After the global decision is executed, execution feedback data is collected to guide multi-level update operations. This feedback data includes action execution accuracy, reflecting the consistency between the action output and the expected target action; language response rationality, measuring the degree of matching between the generated language and the user's expected semantics; and semantic graph matching degree, representing the degree of fit between the nodes and edge relationships in the multimodal semantic graph and the semantic structure of the real task. The update operation first adjusts the parameters of the visual semantic mapping network, language semantic mapping network, and action semantic mapping network individually, using feedback metrics corresponding to each network to drive network weight updates. For example, action execution accuracy drives weight optimization in the action semantic mapping network, and language response rationality drives updates in the language semantic mapping network. Semantic graph construction rules are adjusted based on semantic graph matching degree feedback to optimize node classification and relationship definition, making the graph structure more consistent with the actual task. The weight matrix update of the graph-guided multi-head attention mechanism relies on joint feedback results, using global feedback metrics to comprehensively adjust the attention allocation mode. The meta-learner's learning strategy update includes updating the weights of each task component in the loss function and adjusting the knowledge transfer ratio based on the differences between historical and current tasks. The logic update for partitioning parallel inference subtasks involves redefining the allocation strategy for the corresponding feature dimensions of each subtask based on feedback, such as increasing the priority of feature dimensions related to action execution accuracy in action inference subtasks.

[0085] The parameters of each neural network module can be updated using the backpropagation algorithm, and an adaptive learning rate strategy is employed to adjust the update step size, ensuring stable convergence under the simultaneous influence of multi-dimensional feedback metrics. The visual, language, and action semantic mapping networks each receive updates driven by separate loss branches, with the gradient direction dynamically calculated based on the feedback data weights in each training iteration. The multimodal semantic graph construction rules are updated by dynamically adjusting node category boundary conditions and relation edge threshold definitions to improve graph representation capabilities. In the graph-guided multi-head attention mechanism, the updated weight matrix optimizes the weight distribution through joint feedback, giving higher weights to high-matching regions. The meta-learner learning strategy is updated by adjusting the loss weights of each task and the update frequency of the meta-learning task, enhancing knowledge transfer efficiency. The parallel inference subtask partitioning logic can be adaptively optimized by recalculating the feature importance scores between tasks and dynamically adjusting the task allocation matrix.

[0086] Example Explanation: In the healthcare field, by inputting task-adaptive features into a parallel inference network, the visual inference subtask analyzes key regions of medical images, the language inference subtask parses text descriptions entered by doctors, the action inference subtask processes surgical trajectories, and the cross-modal integrated inference subtask combines these results to provide comprehensive diagnostic suggestions. Subsequently, a gated fusion network integrates the outputs of each subtask to form the final diagnostic decision. After the decision is executed, the execution feedback data includes doctors' assessments of the accuracy of the surgical path recommendation, evaluations of the naturalness and fluency of the patient's interaction with the system, and analysis of the matching degree between the medical knowledge graph and the treatment process. This feedback data drives parameter updates in the visual semantic mapping network, language semantic mapping network, and action semantic mapping network, making the system more accurate and intelligent in subsequent diagnostic tasks, and improving the reliability and adaptability of multimodal diagnostic support.

[0087] In the fintech business domain, task-adaptive features are input into the parallel inference network. The visual inference subtask processes invoice and contract images, the language inference subtask parses customer inquiry texts, and the action inference subtask analyzes user clicks and operation paths. The cross-modal integrated inference subtask combines the results of these subtasks to generate comprehensive financial service recommendations. A gated fusion network weights and fuses the outputs of each subtask to form a global risk assessment or product recommendation decision. Feedback data collected after the global decision execution includes user operation success rate, service language satisfaction, and the business process matching degree of the financial knowledge graph. This data is used to update the parameters of the visual, language, and action semantic mapping networks, adjust the rules for constructing the financial business knowledge graph, and refine the task division logic of the parallel inference subtasks, further enhancing the system's intelligent adaptation and real-time decision-making capabilities for financial products and services across multiple scenarios.

[0088] This embodiment utilizes a dynamic gating fusion mechanism to achieve real-time adjustment of the contribution of each subtask's output, enabling flexible weight distribution of visual, linguistic, action, and cross-modal information in global decision-making based on task context. This approach effectively overcomes the problem of indiscriminate weighting of low-relevance results in traditional static fusion, enhancing the adaptability of decision results to current task requirements and improving the accuracy and robustness of global decision-making.

[0089] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a multimodal semantic graph-guided decision-making method, apparatus, device, and medium, comprising: acquiring visual data, text data, and action data; extracting visual feature vectors, linguistic feature vectors, and action feature vectors respectively; concatenating them to generate initial multimodal features; mapping the visual feature vectors, linguistic feature vectors, and action feature vectors to a shared semantic space to generate shared semantic space features; constructing a multimodal semantic graph based on the initial multimodal features; generating cross-modal alignment features using a graph-guided attention mechanism; generating task-adaptive features by combining the cross-modal alignment features with the task objective input meta-learner; inputting the task-adaptive features into a parallel inference network; executing visual inference sub-tasks, linguistic inference sub-tasks, action inference sub-tasks, and cross-modal integrated inference sub-tasks; generating output results for each sub-task; and integrating the output results of each sub-task through a gating fusion network to generate a global decision. This invention fully explores and aligns deep semantic relationships between different modalities by using a shared semantic space, a graph-guided attention mechanism, and a meta-learner. It also achieves efficient and accurate decision-making in multimodal tasks through parallel inference and a gated fusion network. This invention can solve the problems of insufficient cross-modal semantic relationships and poor adaptability in traditional models, and effectively improve the accuracy and real-time performance in complex multimodal task scenarios.

[0090] In one embodiment, step S10 above includes:

[0091] S101, acquire a visual image or video frame, process the visual image or video frame through a visual feature extraction network, and generate image features containing global context information and local detail information;

[0092] S102, The image features are processed by a semantic coding model to generate an object semantic vector, and a visual feature vector is constructed based on the object semantic vector;

[0093] S103, Obtain the language text, perform word segmentation and part-of-speech tagging on the language text, and generate the segmented and tagged language text;

[0094] S104, Process the segmented and labeled language text through a language coding model to generate an initial language feature representation;

[0095] S105, The initial language feature representation is processed through dependency parsing and semantic role labeling to extract syntactic dependency relations and semantic role information;

[0096] S106, Based on the initial language feature representation, syntactic dependency relations and semantic role information, generate a language feature vector;

[0097] S107, acquire motion sensor time-series data, process the motion sensor time-series data using a time-series feature extraction model, and extract the time-series features of the motion;

[0098] S108, The temporal features of the action are mapped to action semantic concepts through the action semantic mapping table;

[0099] S109, Analyze the action semantic concept through a kinematic model, generate an action semantic feature vector, and construct an action feature vector based on the action semantic feature vector.

[0100] In this embodiment, multimodal features that can be subsequently processed need to be extracted from the multi-source inputs. The visual data source can be a single image or a sequence of video frames, which are input into a visual feature extraction network for processing. The visual feature extraction network typically employs a deep convolutional neural network or a neural architecture with the ability to integrate local receptive fields and multi-scale features, such as a residual network or a visual Transformer. Internally, it can extract contextual features and local detail descriptions at different perceptual scales layer by layer, obtaining an image feature tensor covering overall scene information and key local details. Next, the image feature tensor is input into a semantic encoding model for processing. The semantic encoding model uses fully connected layers and nonlinear activation units to learn to map the feature distribution in the original visual space to a high-dimensional semantic representation space, identify the object categories, attributes, and relative positional relationships in the image, and encode these semantic labels into object semantic vectors. After aggregation, normalization, and embedding transformation, the object semantic vectors finally constitute a complete visual feature vector.

[0101] The text data processing workflow begins with the preprocessing of the original language text. A word segmentation model is applied to divide the continuous text into word or phrase units, and a part-of-speech tagging module is used to label the grammatical category of each unit, such as verb, noun, adjective, etc., for use in downstream grammatical and semantic analysis. The segmented and tagged language text is then input into a language encoding model, such as a language representation network based on recurrent neural networks, Transformer, or BERT structures, to capture contextual dependencies and long-distance dependencies between word sequences, generating an initial language feature representation containing global context. Subsequently, this initial language feature representation is further processed by a dependency parsing module and a semantic role labeling module to parse the subject-verb, verb-object, and modifier dependencies between words in the sentence, as well as the semantic role of each word or phrase in the sentence, such as agent, patient, or instrument. This information is then fused and weighted to obtain a language feature vector containing rich grammatical and semantic relationships.

[0102] Motion data originates from temporal sampling by motion sensors, such as inertial measurement units (IMUs), force sensors, or visual motion capture systems. The raw temporal motion data is input into a temporal feature extraction model, which can be based on one-dimensional convolution, long short-term memory networks (LSTM), temporal Transformers, or self-supervised contrastive learning frameworks. This model learns the dynamic patterns and rhythmic features of the motion in the temporal dimension, extracting physical motion features such as acceleration, angular velocity, and trajectory. The extracted temporal features are semantically labeled using an action semantic mapping table. This table is a predefined set of mapping rules or a trained mapping function, aiming to establish a correspondence between low-level physical motion features and abstract semantic concepts, such as action labels like "rotation" and "extension." The action semantic concepts are then further analyzed by a kinematic model. The kinematic model considers the relationships between spatial position, angle, velocity, and acceleration parameters, calculating the spatial trajectory, temporal continuity, and dynamic consistency of the motion through forward kinematics, inverse kinematics, or trajectory fitting algorithms, ultimately forming an action semantic feature vector that conforms to semantic interpretation. This feature vector is then normalized, scaled, and dimensionally aligned to form a standardized action feature vector.

[0103] Visual feature vectors, linguistic feature vectors, and action feature vectors constitute abstract, high-dimensional, and comparable semantic representations of information from three different sources in multimodal input. By concatenating these three vectors, the modal representations are merged along a specified dimension in the tensor dimension to form initial multimodal features. This concatenation not only preserves the independent semantic representation space within each modality but also provides the basic carrier for cross-modal interaction. The resulting initial multimodal features contain complete semantic descriptions of each modality and facilitate subsequent cross-modal information interaction and fusion through global processing of the overall tensor. The entire process is tightly connected with an end-to-end computational link, ensuring that the raw visual, linguistic, and action data undergo semantic abstraction, spatial alignment, context parsing, and time-series parsing, respectively, and are ultimately fused into a set of initial multimodal features containing global semantics, cross-modal consistency, and local detailed features.

[0104] This embodiment achieves standardized and efficient integration of cross-modal inputs within the same semantic space by performing targeted feature extraction, context modeling, and semantic abstraction mapping on visual, textual, and action data respectively during the multimodal data preprocessing stage, and generating initial multimodal features through vector-level concatenation and fusion. This process ensures the consistency and comparability of semantic expression among visual, linguistic, and action modal data, avoiding the semantic shift problem between modalities caused by directly concatenating original features in traditional methods. Simultaneously, the introduction of fine-grained object semantic vectors, syntactic dependency relations, and action semantic concepts enables the model to capture deep semantic connections inherent in different modalities, improving the expressiveness of multimodal input data and providing a high-quality fusion foundation for subsequent decision-making.

[0105] In one embodiment, step S20 above includes:

[0106] S201, Construct a visual semantic mapping network, a language semantic mapping network, and an action semantic mapping network using a multilayer perceptron structure;

[0107] S202, In the shared semantic space, the visual feature vector is processed by the visual semantic mapping network to generate shared visual semantic features;

[0108] S203, In the shared semantic space, the language feature vector is processed by the language semantic mapping network to generate shared language semantic features;

[0109] S204, In the shared semantic space, the action feature vector is processed by the action semantic mapping network to generate shared action semantic features;

[0110] S205, in the shared semantic space, determine the cross-modal association weights between the shared visual semantic features and the shared linguistic semantic features;

[0111] S206, Adjust the representation distribution of shared action semantic features based on the cross-modal association weights;

[0112] S207, update the embedding representation of the shared visual semantic features and the shared language semantic features using the adjusted shared action semantic features;

[0113] S208 combines the updated shared visual semantic features, the updated shared language semantic features, and the adjusted shared action semantic features into shared semantic space features.

[0114] In this embodiment, by constructing a multilayer perceptron-structured visual semantic mapping network, a language semantic mapping network, and an action semantic mapping network, visual feature vectors, language feature vectors, and action feature vectors can be projected onto a shared semantic space. The multilayer perceptron structure, as a nonlinear mapper, can capture high-order semantic features and complex nonlinear relationships in the input features through a combination of multiple fully connected layers, activation functions, and normalization layers. The input to the visual semantic mapping network is the visual feature vector previously extracted from the visual data. Through layer-by-layer weight transformation, it embeds features in the visual space into a unified semantic embedding space and performs semantic abstraction of local details and global contextual information in the visual feature vectors, outputting shared visual semantic features. The language semantic mapping network processes the language feature vectors. Its network structure can combine embedding layers and context weighting mechanisms to maintain syntactic dependencies and semantic consistency of language features in the shared semantic space, outputting shared language semantic features. Action semantic mapping networks process action feature vectors. The mapping network can contain action-specific time-series modeling units or hidden layer structures adjusted based on kinematic parameters, so that action features can be aligned to a shared semantic space in both spatial and temporal dimensions, and output shared action semantic features.

[0115] In the shared semantic space, it is necessary to further establish the relationships between different modalities. First, this is achieved by calculating the cross-modal association weights between shared visual semantic features and shared linguistic semantic features. These association weights can be calculated using a similarity function (such as dot product similarity or cosine similarity) combined with normalization operations, representing the degree of alignment between visual and linguistic semantic content in the shared space. The calculated association weights are then used to adjust the representation distribution of shared action semantic features. This is achieved by adjusting the semantic clustering or positional information of the action semantic representations in the shared space through linear weighting or gating functions, resulting in higher semantic consistency of the action semantic features within an embedding space consistent with both visual and linguistic features. The adjusted shared action semantic features are then used to update the embedding representations of shared visual and linguistic semantic features. This update process can be implemented through residual connections or interactive attention mechanisms, allowing the introduction of supplementary semantic information aligned with actions into the visual and linguistic embedding representations, thus strengthening the collaborative relationship between the three modalities. Finally, the updated shared visual semantic features, shared language semantic features, and adjusted shared action semantic features are combined together according to tensor concatenation or set structure to form a complete shared semantic space feature, providing a consistent, high-dimensional unified representation for subsequent cross-modal reasoning and interaction.

[0116] This embodiment maps visual feature vectors, linguistic feature vectors, and action feature vectors to a shared semantic space, enabling a unified representation of trimodal data and ensuring direct comparability of data from different modalities within the same space. Cross-modal association weights between visual and linguistic semantic features are calculated and used to adjust action semantic features, effectively mitigating alignment biases caused by significant differences in representations between modalities in traditional multimodal methods. The adjusted action semantic features are then used to update the visual and linguistic semantic embedding representations, resulting in stronger semantic consistency and complementarity among multimodal features. Finally, by combining the updated features, a shared semantic space feature representation is provided that simultaneously maintains the semantic characteristics of each feature and possesses cross-modal collaborative relationships. This provides a unified and high-quality feature foundation for subsequent multimodal interactive reasoning, thereby improving the system's understanding and decision-making accuracy when facing complex multimodal tasks.

[0117] In one embodiment, step S30 above includes:

[0118] S301, Identify key semantic elements from the multimodal initial features;

[0119] S302, the key semantic elements are classified into object nodes, action nodes and result nodes;

[0120] S303, assign visual semantic feature vectors, action semantic feature vectors, and language description semantic feature vectors to the object node, action node, and result node, respectively;

[0121] S304, Establish the tool relationship edge between the object node and the action node;

[0122] S305, Establish the causal relationship edge between the action node and the result node;

[0123] S306 uses the message passing mechanism of graph neural networks to propagate semantic information between object nodes, action nodes and result nodes;

[0124] S307, based on the message passing mechanism, update the feature representations of object nodes, action nodes and result nodes to generate a multimodal semantic graph containing node features and relation edges.

[0125] In this embodiment, the multimodal initial features serve as the input source, containing comprehensive information extracted and concatenated from visual, linguistic, and action data. First, key semantic elements are identified from the multimodal initial features. This identification process can be achieved through rule-based entity extraction or deep learning models (such as sequence labeling networks) to jointly analyze the combined features of visual, textual, and action data to determine elements with independent semantic meaning, such as specific objects, actions performed, or task results. The identified key semantic elements need to be classified into object nodes, action nodes, and result nodes. Each type of node represents a different semantic role: object nodes correspond to perceptible entities in the multimodal data, action nodes correspond to operations expressed in time series or descriptions, and result nodes correspond to the expected operation result or task output.

[0126] For each node type, a corresponding semantic vector needs to be assigned. Object nodes are assigned visual semantic feature vectors, which are obtained by extracting features from the object image using a visual encoder. Action nodes are assigned action semantic feature vectors, which can be obtained by analyzing motion sensor data or embedding representations of action description statements. Result nodes are assigned linguistic description semantic feature vectors, which are obtained by encoding the task objective or expected result description using a language model. The assignment of node features establishes a direct connection between the node itself and the different modal data in the multimodal initial features, forming the basic graph structure node information.

[0127] Relational modeling introduces structured edge representations into graph structures. Tool relationship edges are established between object nodes and action nodes. These edges represent the association between an object as a tool or operation object and an action, such as the tool usage relationship between "compass" and "drawing a circle". Causal relationship edges are established between action nodes and result nodes. These edges represent the causal dependency between actions and results, such as the relationship between "drawing a circle" and "generating a circular pattern".

[0128] After defining nodes and relationships in the graph structure, semantic information is propagated among object nodes, action nodes, and result nodes through the message passing mechanism of a graph neural network. This mechanism encodes the feature information of neighboring nodes into the representation of the current node using multi-layer graph convolution, weighted aggregation, and non-linear activation functions, thus globally modeling the contextual dependencies between different nodes. Through multiple iterative propagation, the feature representation of a node gradually incorporates the semantic context of its neighboring nodes and their relationships. Based on the message passing mechanism, the feature representations of object nodes, action nodes, and result nodes are finally updated. The updated node features, along with relationship edges, form a complete multimodal semantic graph containing node features and relationship edges, which is then used for subsequent attention mechanisms and inference processing.

[0129] This embodiment constructs a multimodal semantic graph based on initial multimodal features, enabling explicit modeling of element relationships between different modalities. This allows object nodes, action nodes, and result nodes to be connected within a unified graph structure through visual, linguistic, and action features. The definition of tool-related edges and causal-related edges enhances the semantic connections between different nodes, and the message passing mechanism of the graph neural network efficiently propagates and integrates multimodal contextual information between nodes. The final updated multimodal semantic graph not only retains the independent expression of each modality's features but also reflects cross-modal semantic associations and contextual structure, enabling subsequent processing stages to utilize globally consistent semantic relationship information to improve the accuracy of multimodal understanding and task decision-making.

[0130] In one embodiment, step S40 above includes:

[0131] S401, obtain the feature element vector in the initial features of the multimodal system and the node feature vector in the multimodal semantic graph;

[0132] S402, determine the similarity score between the feature element vector and the node feature vector, process the similarity score through an activation function, and generate a correlation weight value;

[0133] S403, Normalize the correlation weights to generate a standardized attention weight distribution;

[0134] S404, The multimodal initial features are weighted and aggregated based on the standardized attention weight distribution to generate cross-modal aligned features.

[0135] In this embodiment, the multimodal initial features are a concatenated vector set integrating visual, linguistic, and action modalities. They contain feature element vectors corresponding to different modalities. These feature element vectors are low-dimensional or high-dimensional vector representations extracted from the input data. Visual feature elements can originate from visual region representations encoded by convolutional neural networks; linguistic feature elements can originate from word or phrase embedding vectors from language models; and action feature elements can originate from feature encoding results of action sensor data from temporal analysis models. The multimodal semantic graph contains node feature vectors. These node feature vectors, updated through a graph neural network, encode the semantic information and contextual relationships of object nodes, action nodes, and result nodes.

[0136] After obtaining the feature element vectors from the initial multimodal features and the node feature vectors from the multimodal semantic graph, they need to be matched. Various metrics can be used to determine the similarity score between the feature element vectors and the node feature vectors, such as dot product calculation, cosine similarity, or a trainable attention scoring function. The similarity score measures the relevance between the input feature elements and the graph nodes in the shared semantic space. The application of activation functions enhances the numerical discriminative power of the similarity score through non-linear mapping; commonly used activation functions include LeakyReLU or the Sigmoid function.

[0137] The relevance weights need to be normalized to generate a standardized attention weight distribution. Normalization is typically performed using a softmax function, ensuring that the sum of all relevance weights is 1. This facilitates interpretation as a probability distribution or importance weight in weighted aggregation. The standardized attention weight distribution highlights highly relevant matches and suppresses low-relevance matches, enabling the attention mechanism to assign higher weights to more relevant features in feature aggregation.

[0138] Based on a standardized attention weight distribution, weighted aggregation of multimodal initial features is performed, which means summing each feature element vector according to its attention weight with the feature vectors of each node. This aggregation process can be implemented using matrix multiplication or weighted summation, so that the initial features of different modalities form a comprehensive vector representation after aggregation. The generated cross-modal aligned features have cross-modal consistency, graph context relevance, and semantic consistency, serving as an important input for subsequent adaptive processing.

[0139] This embodiment utilizes an attention mechanism guided by multimodal initial features and a multimodal semantic graph input graph to establish fine-grained correspondences between different modalities and dynamically adjust the fusion ratio of information from each modality. Similarity calculation and activation processing ensure that only semantically related elements are focused on, and the normalized attention weight distribution makes the feature fusion process interpretable and controllable. The final weighted aggregation ensures that cross-modal aligned features have global semantic consistency, which can improve the accuracy of task understanding and decision-making in multimodal scenarios, while reducing redundant information interference and improving the model's robustness to complex multimodal inputs.

[0140] In one embodiment, step S50 above includes:

[0141] S501, encode the task objective into a task objective vector;

[0142] S502, input the cross-modal alignment features and the task target vector into the meta-learner;

[0143] S503, extract common knowledge representations among different tasks through the meta-learner;

[0144] S504, identify the differences between the current task and historical tasks;

[0145] S505, obtain the initial parameter matrix of the meta-learner;

[0146] S506, Based on the common knowledge representation, determine the parameter space distribution characteristics, and extract key feature dimensions from the difference characteristics;

[0147] S507, Based on the parameter space distribution characteristics and key feature dimensions, the initial parameter matrix is ​​optimized by the gradient descent module to generate a transferable model parameter matrix;

[0148] S508, adjust the feature transformation matrix using the transferable model parameter matrix to generate the adjusted feature transformation matrix;

[0149] S509, The cross-modal alignment features are processed by the adjusted feature transformation matrix to generate task-adaptive features.

[0150] In this embodiment, the cross-modal alignment feature is a unified representation of visual, linguistic, and action modalities after graph-guided attention aggregation. The task objective is a high-level semantic information describing the current task requirements. When the task objective is encoded as a task objective vector, a pre-trained text encoder can be used to convert the task description into a dense vector form for computation. The cross-modal alignment feature and the task objective vector are input together into a meta-learner. The meta-learner, as a neural network structure for task adaptation, supports rapid parameter adjustment to adapt to different tasks using a small number of training samples.

[0151] The meta-learner first extracts common knowledge representations across different tasks. These common knowledge representations can be embodied in the shared parameter set learned during multi-task training, with the aim of capturing the prevalent regularities across different tasks. When identifying the differences between the current task and historical tasks, it is necessary to compare the distribution differences between the current task input and historical tasks, such as through statistical analysis based on the mean and covariance matrices or feature differentiation representation based on the difference encoding module.

[0152] The initial parameter matrix of the meta-learner can be derived from the parameter set obtained during the pre-training stage. This initial parameter matrix contains multi-dimensional weights and biases, serving as the starting point for subsequent parameter adjustments. Based on common knowledge representation, the parameter space distribution characteristics are determined, representing the statistical distribution of general task patterns in the parameter space. For example, the parameter distribution learned from historical tasks can be fitted using a Gaussian distribution or variational inference model. Simultaneously, key feature dimensions are extracted from the differential features. The extraction of key feature dimensions can reduce dimensionality and highlight task-specific information through attention mechanisms or principal component analysis.

[0153] The gradient descent module optimizes the initial parameter matrix by minimizing the current task's loss function, generating a transferable model parameter matrix. This optimization process combines common knowledge distributions with the unique characteristics of the current task, ensuring that the optimized parameters balance task generality and adaptability. The transferable model parameter matrix is ​​used to adjust the feature transformation matrix, which controls the projection and linear transformation of cross-modal aligned features across dimensions. The adjusted feature transformation matrix better matches the feature distribution of the current task.

[0154] Finally, the cross-modal aligned features are processed by the adjusted feature transformation matrix. The output task-adaptive features are multimodal comprehensive representations with good transferability and adaptability, which are optimized for the current task requirements based on the shared semantic space and serve as inputs for subsequent decision-making or reasoning modules.

[0155] This embodiment inputs cross-modal alignment features and task objectives into the meta-learner, and optimizes the parameter matrix by combining common knowledge representation, differential features, and parameter space distribution. This enables effective adjustment of model parameters in a small number of samples or rapidly changing tasks, allowing task-adaptive features to accurately align with the current task requirements in spatial representation. This improves the ability to quickly transfer between different tasks, reduces retraining costs, and enhances the model's versatility and flexibility in multi-task environments.

[0156] In one embodiment, step S60 above includes:

[0157] S601, Assign the feature dimensions related to visual objects in the task adaptability features to the visual reasoning subtask;

[0158] S602, Assign the feature dimensions related to semantic parsing in the task adaptability features to the language reasoning subtask;

[0159] S603, Assign the feature dimensions related to the action trajectory in the task adaptability features to the action reasoning subtask;

[0160] S604, Assign the feature dimensions related to multimodal decision-making in the task adaptability features to the cross-modal integrated reasoning subtask;

[0161] S605, The visual reasoning subtask, language reasoning subtask, action reasoning subtask and cross-modal integrated reasoning subtask are executed synchronously in a parallel processing environment.

[0162] S606, The visual reasoning subtask processes the feature dimensions related to the visual object and generates a visual reasoning result;

[0163] S607, The language reasoning subtask processes the feature dimensions related to semantic parsing and generates language reasoning results;

[0164] S608, The action reasoning subtask processes the feature dimensions related to the action trajectory and generates an action reasoning result;

[0165] S609, The feature dimensions related to multimodal decision-making are processed through the cross-modal integrated reasoning subtask to generate cross-modal integrated reasoning results.

[0166] In this embodiment, task-adaptive features are multimodal comprehensive feature representations that have undergone feature adjustment and optimization within a shared semantic space to meet the current task requirements, and possess distribution attributes strongly correlated with the task scenario. Inputting task-adaptive features into the parallel inference network requires first parsing their internal distribution, allocating the feature dimensions relevant to visual objects to the visual inference subtask. This allocation can be accomplished through a predefined feature dimension index or a dynamic routing algorithm based on feature importance scores, ensuring that information directly related to visual tasks such as image object detection and recognition is specifically passed to the visual inference subtask.

[0167] When assigning semantic parsing-related feature dimensions from task-adaptive features to the language reasoning subtask, it is necessary to extract vector components that are strongly associated with language context, grammatical structure, and vocabulary from the task-adaptive features. This can be done by combining the weights of vocabulary-related dimensions in the embedding representation or the results of dependency parsing. When assigning action trajectory-related feature dimensions to the action reasoning subtask, it is necessary to identify those vector dimensions that encode motion trajectories, dynamic postures, and temporal patterns to ensure that the feature information input to the action reasoning subtask can support motion prediction and behavior analysis.

[0168] When assigning feature dimensions related to multimodal decision-making to the cross-modal integrated reasoning subtask, it is necessary to analyze the parts of the task-adaptive features that have the highest interaction relationship with multimodal processes and decision-making relevance. This typically involves higher-order features after fusion. This assignment can be done based on global contextual dependencies or higher-order connectivity relationships of semantic graph nodes.

[0169] After allocation, the parallel inference network needs to execute four types of inference subtasks synchronously in a parallel computing environment. The parallel computing environment can be implemented through multi-threaded scheduling, GPU multi-stream concurrency, or heterogeneous multi-core architecture to ensure that the visual inference subtask, language inference subtask, action inference subtask, and cross-modal synthesis inference subtask can be started and run synchronously at the same time, so as to reduce the overall inference latency.

[0170] The visual reasoning subtask processes feature dimensions related to visual objects and outputs visual reasoning results through convolutional neural networks or a visual Transformer module. The language reasoning subtask processes feature dimensions related to semantic parsing using a Transformer or recurrent neural network to generate language reasoning results. The action reasoning subtask uses temporal modeling structures (e.g., Long Short-Term Memory networks or temporal Transformers) to process feature dimensions related to action trajectories and generate action reasoning results. The cross-modal comprehensive reasoning subtask processes feature dimensions related to multimodal decision-making through a cross-modal Transformer or a multimodal graphical convolutional network to generate cross-modal comprehensive reasoning results. All outputs serve as input to the subsequent global decision-making module.

[0171] Example Description: In an intelligent healthcare assistance system, visual, textual, and motion data from the medical scenario are first acquired through a multimodal perception module. Visual data includes patient facial images, limb photographs, and medical examination images, such as X-rays and skin lesion images, captured by high-definition cameras. Textual data includes patient-provided symptom descriptions, medical records, and electronic medical record content, input from electronic medical records or a speech-to-text module. Motion data records key actions of doctors or patients during diagnosis and treatment, such as gestures during surgical simulation training or patient movement trajectories during rehabilitation training, continuously acquired through motion sensors (e.g., inertial measurement units, IMUs).

[0172] After data acquisition, the visual feature extraction network processes medical images, extracting global contextual information such as the overall outline of human body parts and local details such as skin spots and abnormal limbs. It further generates object semantic vectors through a semantic coding model, such as semantic labels for specific disease symptoms. These semantic vectors are integrated into visual feature vectors. For text data, word segmentation and part-of-speech tagging are performed first, followed by initial linguistic feature representations obtained through a language coding model. Dependency parsing and semantic role labeling are then performed to extract semantic relationships such as patient complaints, accompanying symptoms, and disease progression, ultimately forming linguistic feature vectors. Action data is analyzed by a temporal feature extraction model to determine parameters such as action amplitude, speed, and period. An action semantic mapping table maps action sequences to semantic concepts such as "decreased stability" or "slowed movement," and a kinematic model calculates the action semantic feature vectors.

[0173] Visual feature vectors, linguistic feature vectors, and action feature vectors are concatenated to form multimodal initial features, which are then mapped to a shared semantic space. In this process, a visual semantic mapping network extracts shared visual semantic features related to medical images, a linguistic semantic mapping network extracts shared linguistic semantic features related to patient complaints, and an action semantic mapping network extracts shared action semantic features related to patient behavior. Within the shared semantic space, cross-modal association weights between shared visual and linguistic semantic features are determined, such as the correlation between a patient's description of a "rash" and an image of detected skin abnormalities. Based on this, the distribution of shared action semantic features is adjusted; for example, when both text and image point to a movement disorder, the expression of relevant action features is strengthened. Subsequently, the adjusted action semantic features are used to optimize the embedding representation of visual and linguistic features, and finally, all updated features are combined into shared semantic space features.

[0174] The system constructs a multimodal semantic graph based on initial multimodal features, identifying key semantic elements such as "limb abnormalities," "gait instability," and "medical records," classifying them into object nodes (e.g., abnormal areas in images), action nodes (e.g., gait abnormalities), and outcome nodes (e.g., fall risk). These nodes are assigned visual semantic feature vectors, action semantic feature vectors, and linguistic description semantic feature vectors, respectively, and tool-related edges (e.g., the association between "gait abnormalities" and "knee joint images") and causal-related edges (e.g., the causal relationship between "fall risk" and "gait instability") are established. Through the message passing mechanism of a graph neural network, multimodal semantic information is propagated between nodes, dynamically updating node feature representations and forming a structured multimodal semantic graph.

[0175] A multimodal initial feature and a multimodal semantic graph are introduced using an attention mechanism guided by the input graph. Similarity between initial feature elements and graph node features is calculated and processed using an activation function to generate correlation weights. These weights are then normalized to form a standardized attention weight distribution. Initial features are aggregated based on this distribution to ultimately generate aligned cross-modal features, ensuring a high degree of information consistency across different modalities.

[0176] Cross-modal alignment features and the task objective (e.g., "identifying patients at high fall risk") are jointly input into the meta-learner. The meta-learner first encodes the task objective as a vector, then extracts common knowledge representations across different tasks (e.g., prevalent patterns of fall risk in different patient groups) and identifies discrepancies between the current patient and historical patients (e.g., personalized gait features of a specific patient). Based on the common knowledge, the parameter space distribution features are determined, key feature dimensions are extracted from the discrepancies, and gradient descent is used to optimize the initial parameter matrix of the meta-learner, generating a transferable model parameter matrix. This matrix is ​​used to adjust the feature transformation matrix and ultimately to process the cross-modal alignment features, outputting task-adaptive features that fit the characteristics of the current patient.

[0177] Task-adaptive features are input into a parallel inference network. Through preset rules or dynamic allocation strategies, the system assigns feature dimensions related to visual objects to visual inference subtasks, handling visual tasks related to abnormal region detection in images; feature dimensions related to semantic parsing to language inference subtasks, handling tasks related to medical record text analysis; feature dimensions related to motion trajectories to motion inference subtasks, handling tasks related to patient gait analysis; and feature dimensions related to multimodal decision-making to cross-modal comprehensive inference subtasks, handling overall health risk prediction. These four types of subtasks are executed synchronously in a parallel computing environment, outputting visual inference results (e.g., locating abnormal image regions), language inference results (e.g., extracting major symptoms and complications), motion inference results (e.g., gait stability scores), and cross-modal comprehensive inference results (e.g., comprehensive health risk scores), respectively.

[0178] Finally, the outputs of each subtask are integrated through a gating fusion network. The gating unit dynamically assigns fusion weights based on the importance of various inference results. For example, in gait abnormality scenarios, the weight of action inference results is increased, and in scenarios with insufficient language description, the weight of visual inference results is increased. After weighted fusion, a global decision instruction is generated, such as "The patient has a high risk of falling; rehabilitation intervention training and imaging follow-up are recommended."

[0179] In the intelligent financial customer service and risk analysis system, visual, text, and motion data are first collected through a multimodal perception module. Visual data includes images of customers' ID cards, photos of business documents, and facial expressions, captured in real-time by high-definition cameras. Text data includes the content of completed business application forms, contract terms, and financial product descriptions, directly obtained by the electronic form system. Motion data records customer interactions during self-service terminal or counter operations, such as gesture selection, operation sequence, and pause times, collected in real-time by touchscreens and interactive sensors.

[0180] In financial scenarios, visual feature extraction networks process image data, extracting global contextual information such as the global boundaries of an ID card and local details such as signature areas and seal information. These image features are then transformed into semantic vectors related to identity authentication and document integrity through a semantic encoding model, forming visual feature vectors. Text data undergoes word segmentation and part-of-speech tagging, using a language encoding model to obtain initial linguistic feature representations. Dependency parsing and semantic role labeling are then used to extract the rights and responsibilities between contract terms, the constraints between financial products, and temporal conditions, generating linguistic feature vectors. Action data is analyzed by a temporal feature extraction model, such as the click trajectory, swipe path, and dwell time of customers filling out forms on a terminal. Combined with an action semantic mapping table, these behaviors are transformed into risk warning semantics, such as "abnormal pause" or "rapid item skipping." A kinematic model is then used to generate action semantic feature vectors related to risky behaviors, constituting the action feature vectors.

[0181] Visual, linguistic, and action feature vectors are concatenated to form multimodal initial features, which are then mapped to a shared semantic space. The visual semantic mapping network processes visual features related to identity information and credential content; the linguistic semantic mapping network processes linguistic features related to contract terms and form descriptions; and the action semantic mapping network processes action features related to customer operating habits and behavioral risks. In the shared semantic space, cross-modal association weights between visual and linguistic features are calculated, such as the consistency between the contract text "ID card signature" and the "signature area" in the image, to adjust the representation distribution of action features. For example, if an operational anomaly is likely strongly correlated with a certain field in a form, that action feature is strengthened. Through this adjustment, the embedded representations of visual and linguistic features are further updated, and finally fused to generate shared semantic space features.

[0182] Based on the initial multimodal features, the system constructs a multimodal semantic graph to identify key semantic elements such as "high-amount transfers," "abnormal operations," and "contract defects." These elements are classified into object nodes (image data), action nodes (customer interaction behaviors), and result nodes (potential risk events). Visual, action, and linguistic semantic vectors are assigned to the nodes respectively, establishing tool relationship edges such as "contract image and identity document" and causal relationship edges such as "abnormal operations and high-risk events." The graph neural network propagates this multimodal semantic information between nodes, updating the node feature representations in real time to form a structured multimodal semantic graph.

[0183] An attention mechanism guided by the input graph is implemented, involving multimodal initial features and a multimodal semantic graph. Similarity scores are calculated between initial feature elements and semantic graph node features, and activation functions are used to generate association weights. After normalization, a standardized attention weight distribution is formed. Based on this weight distribution, the multimodal initial features are weighted and aggregated to generate aligned cross-modal features, ensuring high consistency between customer identity, business application content, and risky behavior in the multimodal space.

[0184] Cross-modal alignment features and the task objective (e.g., "reviewing high-amount transfer applications") are input into the meta-learner. The meta-learner first encodes the task objective into a task objective vector, extracts common knowledge prevalent in different financial transactions, such as the correlation pattern between customer identity consistency and contract integrity, and identifies the differences between the current customer and historical transactions, such as the specific customer's operating habits and preferences. Combining common knowledge and differences, the parameter space distribution is calculated, key feature dimensions are extracted, the initial parameter matrix is ​​optimized, and a transferable model parameter matrix is ​​generated to adjust the feature transformation matrix. The adjusted feature transformation matrix processes the cross-modal alignment features and outputs task-adaptive features suitable for the current task and the corresponding customer.

[0185] Task-adaptive features are input into a parallel inference network. In this network, visual object-related features are assigned to visual inference subtasks, such as verifying the integrity of ID card information in customer images; semantic parsing-related features are assigned to linguistic inference subtasks, such as reviewing the legality of contract terms; motion trajectory-related features are assigned to motion inference subtasks, such as detecting abnormal patterns in customer input behavior; and multimodal decision-making-related features are assigned to cross-modal comprehensive inference subtasks, such as comprehensively assessing the consistency and risk between customer behavior and identity information and contract terms. These four types of subtasks are computed in parallel and output visual inference results (e.g., ID card image integrity score), linguistic inference results (e.g., contract term defect detection results), motion inference results (e.g., abnormal interaction probability), and cross-modal comprehensive inference results (e.g., comprehensive score for high-risk business applications).

[0186] Finally, the output results of each subtask are integrated through a gating fusion network. The gating unit dynamically assigns fusion weights, for example, emphasizing language reasoning results in complex contract scenarios, and emphasizing action and language subtask results when the customer image is unclear. After weighted fusion, a global decision is output, such as "This customer has a high risk of operational anomalies, manual review is recommended" or "The contract terms of this application have key inconsistencies, requiring secondary verification."

[0187] This embodiment precisely divides task-adaptive features into four sub-tasks: vision, language, action, and multimodal integration. The parallel inference network can utilize parallel computing resources to process various types of information synchronously, fully leveraging the independence of different sub-tasks and reducing overall inference latency. The precise allocation of feature dimensions ensures that each sub-task receives the most relevant information input, improving inference efficiency and accuracy. Especially in multimodal task concurrency scenarios, it achieves fast and accurate multimodal inference and decision-making.

[0188] In one embodiment, a multimodal semantic graph-guided decision-making device is provided, which corresponds one-to-one with the multimodal semantic graph-guided decision-making method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multimodal semantic graph-guided decision-making device of the present invention. The modules include a multimodal feature extraction module 10, a semantic mapping module 20, a semantic graph construction module 30, a graph-guided attention module 40, a task adaptation module 50, a parallel reasoning module 60, and a decision fusion module 70. Detailed descriptions of each functional module are as follows:

[0189] The multimodal feature extraction module 10 is used to acquire visual data, text data and action data, extract visual feature vectors, language feature vectors and action feature vectors from the visual data, text data and action data respectively, and concatenate the visual feature vectors, language feature vectors and action feature vectors to generate multimodal initial features;

[0190] Semantic mapping module 20 is used to map the visual feature vector, language feature vector and action feature vector to the shared semantic space respectively, and generate shared semantic space features;

[0191] Semantic graph construction module 30 is used to construct a multimodal semantic graph based on the multimodal initial features;

[0192] The graph-guided attention module 40 is used to input the multimodal initial features and the multimodal semantic graph into the graph-guided attention mechanism to generate cross-modal aligned features;

[0193] The task adaptation module 50 is used to combine the cross-modal alignment features with the task target input meta-learner to generate task-adaptive features;

[0194] Parallel reasoning module 60 is used to input the task adaptive features into the parallel reasoning network, and execute visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks and cross-modal integrated reasoning sub-tasks in parallel in the parallel reasoning network, so as to process the feature content corresponding to each sub-task in the task adaptive features and generate the output results of each sub-task.

[0195] The decision fusion module 70 is used to integrate the output results of each subtask through a gating fusion network to generate a global decision.

[0196] In one embodiment, the multimodal feature extraction module 10 is specifically used for:

[0197] Acquire visual images or video frames, process the visual images or video frames through a visual feature extraction network, and generate image features containing global context information and local detail information;

[0198] The image features are processed by a semantic coding model to generate object semantic vectors, and visual feature vectors are constructed based on the object semantic vectors.

[0199] Obtain the language text, perform word segmentation and part-of-speech tagging on the language text, and generate the segmented and tagged language text;

[0200] The segmented and annotated language text is processed using a language coding model to generate an initial language feature representation.

[0201] The initial language feature representation is processed by dependency parsing and semantic role labeling to extract syntactic dependency relations and semantic role information;

[0202] Based on the initial language feature representation, syntactic dependency relations, and semantic role information, a language feature vector is generated;

[0203] Acquire motion sensor time-series data, process the motion sensor time-series data using a time-series feature extraction model, and extract the time-series features of the motion.

[0204] The temporal features of the actions are mapped to action semantic concepts through an action semantic mapping table;

[0205] The action semantic concept is analyzed by kinematic model, and action semantic feature vectors are generated. Action feature vectors are then constructed based on the action semantic feature vectors.

[0206] In one embodiment, the semantic mapping module 20 is specifically used for:

[0207] Construct a visual semantic mapping network, a language semantic mapping network, and an action semantic mapping network using a multilayer perceptron structure;

[0208] In the shared semantic space, the visual feature vectors are processed by the visual semantic mapping network to generate shared visual semantic features;

[0209] In the shared semantic space, the language feature vectors are processed by the language semantic mapping network to generate shared language semantic features;

[0210] In the shared semantic space, the action feature vector is processed by the action semantic mapping network to generate shared action semantic features;

[0211] In the shared semantic space, determine the cross-modal association weights between the shared visual semantic features and the shared linguistic semantic features;

[0212] The representation distribution of shared action semantic features is adjusted based on the cross-modal association weights;

[0213] The embedded representations of the shared visual semantic features and shared linguistic semantic features are updated using the adjusted shared action semantic features;

[0214] The updated shared visual semantic features, the updated shared language semantic features, and the adjusted shared action semantic features are combined into shared semantic space features.

[0215] In one embodiment, the semantic graph construction module 30 is specifically used for:

[0216] Identify key semantic elements from the multimodal initial features;

[0217] The key semantic elements are classified into object nodes, action nodes, and result nodes.

[0218] Assign visual semantic feature vectors, action semantic feature vectors, and language description semantic feature vectors to the object nodes, action nodes, and result nodes, respectively;

[0219] Establish tool relationship edges between the object nodes and action nodes;

[0220] Establish causal relationship edges between the action nodes and result nodes;

[0221] Semantic information is propagated between object nodes, action nodes, and result nodes through the message passing mechanism of graph neural networks;

[0222] Based on the aforementioned message passing mechanism, the feature representations of object nodes, action nodes, and result nodes are updated to generate a multimodal semantic graph containing node features and relational edges.

[0223] In one embodiment, the graph-guided attention module 40 is specifically used for:

[0224] Obtain the feature element vectors from the initial multimodal features and the node feature vectors from the multimodal semantic graph;

[0225] The similarity score between the feature element vector and the node feature vector is determined, and the similarity score is processed by an activation function to generate a correlation weight value.

[0226] The correlation weights are normalized to generate a standardized attention weight distribution;

[0227] The multimodal initial features are weighted and aggregated based on the standardized attention weight distribution to generate cross-modal aligned features.

[0228] In one embodiment, the task adaptation module 50 is specifically used for:

[0229] Encode the task objective into a task objective vector;

[0230] The cross-modal alignment features and the task target vector are input into the meta-learner;

[0231] The meta-learner extracts common knowledge representations across different tasks;

[0232] Identify the differences between the current task and historical tasks;

[0233] Obtain the initial parameter matrix of the meta-learner;

[0234] Based on the common knowledge representation, the spatial distribution characteristics of the parameters are determined, and key feature dimensions are extracted from the differential characteristics;

[0235] Based on the parameter space distribution characteristics and key feature dimensions, the initial parameter matrix is ​​optimized by the gradient descent module to generate a transferable model parameter matrix.

[0236] The feature transformation matrix is ​​adjusted using the transferable model parameter matrix to generate the adjusted feature transformation matrix.

[0237] The cross-modal alignment features are processed by the adjusted feature transformation matrix to generate task-adaptive features.

[0238] In one embodiment, the parallel inference module 60 is specifically used for:

[0239] Assign the feature dimensions related to visual objects from the task adaptability features to the visual reasoning subtask;

[0240] Assign the feature dimensions related to semantic parsing from the task-adaptive features to the language reasoning subtask;

[0241] Assign the feature dimensions related to action trajectory from the task adaptability features to the action reasoning subtask;

[0242] Assign the feature dimensions related to multimodal decision-making from the aforementioned task adaptability features to the cross-modal integrated reasoning subtask;

[0243] The visual reasoning subtask, language reasoning subtask, action reasoning subtask, and cross-modal integrated reasoning subtask are executed synchronously in a parallel processing environment.

[0244] The visual reasoning subtask processes the feature dimensions related to the visual object to generate a visual reasoning result.

[0245] The language reasoning subtask processes the feature dimensions related to semantic parsing to generate language reasoning results;

[0246] The action reasoning subtask processes the feature dimensions related to the action trajectory to generate action reasoning results;

[0247] The cross-modal integrated reasoning subtask processes the feature dimensions related to multimodal decision-making to generate cross-modal integrated reasoning results.

[0248] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal semantic graph-guided decision-making method on the server side.

[0249] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides decision-making and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements user-side functions or steps of a multimodal semantic graph-guided decision-making method.

[0250] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0251] Acquire visual data, text data, and action data; extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively; concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features;

[0252] The visual feature vector, language feature vector, and action feature vector are respectively mapped to a shared semantic space to generate shared semantic space features;

[0253] A multimodal semantic graph is constructed based on the aforementioned multimodal initial features;

[0254] The initial multimodal features and the multimodal semantic graph are input into a graph-guided attention mechanism to generate cross-modal aligned features;

[0255] The cross-modal alignment features and the task objective are input into the meta-learner to generate task-adaptive features;

[0256] The task-adaptive features are input into a parallel inference network, in which visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks, and cross-modal integrated reasoning sub-tasks are executed in parallel to process the feature content corresponding to each sub-task in the task-adaptive features and generate the output results of each sub-task.

[0257] The output results of each subtask are integrated through a gated fusion network to generate a global decision.

[0258] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0259] Acquire visual data, text data, and action data; extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively; concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features;

[0260] The visual feature vector, language feature vector, and action feature vector are respectively mapped to a shared semantic space to generate shared semantic space features;

[0261] A multimodal semantic graph is constructed based on the aforementioned multimodal initial features;

[0262] The initial multimodal features and the multimodal semantic graph are input into a graph-guided attention mechanism to generate cross-modal aligned features;

[0263] The cross-modal alignment features and the task objective are input into the meta-learner to generate task-adaptive features;

[0264] The task-adaptive features are input into a parallel inference network, in which visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks, and cross-modal integrated reasoning sub-tasks are executed in parallel to process the feature content corresponding to each sub-task in the task-adaptive features and generate the output results of each sub-task.

[0265] The output results of each subtask are integrated through a gated fusion network to generate a global decision.

[0266] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0267] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0268] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0269] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A decision-making method guided by a multimodal semantic graph, characterized in that, Includes the following steps: Acquire visual data, text data, and action data; extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively; concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features; The visual feature vector, language feature vector, and action feature vector are respectively mapped to a shared semantic space to generate shared semantic space features; A multimodal semantic graph is constructed based on the aforementioned multimodal initial features; The initial multimodal features and the multimodal semantic graph are input into a graph-guided attention mechanism to generate cross-modal aligned features; The cross-modal alignment features and the task objective are input into the meta-learner to generate task-adaptive features; The task-adaptive features are input into a parallel inference network, in which visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks, and cross-modal integrated reasoning sub-tasks are executed in parallel to process the feature content corresponding to each sub-task in the task-adaptive features and generate the output results of each sub-task. The output results of each subtask are integrated through a gated fusion network to generate a global decision.

2. The decision-making method guided by multimodal semantic graphs as described in claim 1, characterized in that, Acquire visual data, text data, and action data; extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively; concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features, including: Acquire visual images or video frames, process the visual images or video frames through a visual feature extraction network, and generate image features containing global context information and local detail information; The image features are processed by a semantic coding model to generate object semantic vectors, and visual feature vectors are constructed based on the object semantic vectors. Obtain the language text, perform word segmentation and part-of-speech tagging on the language text, and generate the segmented and tagged language text; The segmented and annotated language text is processed using a language coding model to generate an initial language feature representation. The initial language feature representation is processed by dependency parsing and semantic role labeling to extract syntactic dependency relations and semantic role information; Based on the initial language feature representation, syntactic dependency relations, and semantic role information, a language feature vector is generated; Acquire motion sensor time-series data, process the motion sensor time-series data using a time-series feature extraction model, and extract the time-series features of the motion. The temporal features of the actions are mapped to action semantic concepts through an action semantic mapping table; The action semantic concept is analyzed by kinematic model, and action semantic feature vectors are generated. Action feature vectors are then constructed based on the action semantic feature vectors.

3. The decision-making method guided by multimodal semantic graphs as described in claim 1, characterized in that, The visual feature vector, language feature vector, and action feature vector are mapped to a shared semantic space to generate shared semantic space features, including: Construct a visual semantic mapping network, a language semantic mapping network, and an action semantic mapping network using a multilayer perceptron structure; In the shared semantic space, the visual feature vectors are processed by the visual semantic mapping network to generate shared visual semantic features; In the shared semantic space, the language feature vectors are processed by the language semantic mapping network to generate shared language semantic features; In the shared semantic space, the action feature vector is processed by the action semantic mapping network to generate shared action semantic features; In the shared semantic space, determine the cross-modal association weights between the shared visual semantic features and the shared linguistic semantic features; The representation distribution of shared action semantic features is adjusted based on the cross-modal association weights; The embedded representations of the shared visual semantic features and shared linguistic semantic features are updated using the adjusted shared action semantic features; The updated shared visual semantic features, the updated shared language semantic features, and the adjusted shared action semantic features are combined into shared semantic space features.

4. The decision-making method guided by multimodal semantic graphs as described in claim 1, characterized in that, Constructing a multimodal semantic graph based on the aforementioned multimodal initial features includes: Identify key semantic elements from the multimodal initial features; The key semantic elements are classified into object nodes, action nodes, and result nodes. Assign visual semantic feature vectors, action semantic feature vectors, and language description semantic feature vectors to the object nodes, action nodes, and result nodes, respectively; Establish tool relationship edges between the object nodes and action nodes; Establish causal relationship edges between the action nodes and result nodes; Semantic information is propagated between object nodes, action nodes, and result nodes through the message passing mechanism of graph neural networks; Based on the aforementioned message passing mechanism, the feature representations of object nodes, action nodes, and result nodes are updated to generate a multimodal semantic graph containing node features and relational edges.

5. The decision-making method guided by multimodal semantic graphs as described in claim 1, characterized in that, The multimodal initial features and the multimodal semantic graph are input into a graph-guided attention mechanism to generate cross-modal aligned features, including: Obtain the feature element vectors from the initial multimodal features and the node feature vectors from the multimodal semantic graph; The similarity score between the feature element vector and the node feature vector is determined, and the similarity score is processed by an activation function to generate a correlation weight value. The correlation weights are normalized to generate a standardized attention weight distribution; The multimodal initial features are weighted and aggregated based on the standardized attention weight distribution to generate cross-modal aligned features.

6. The decision-making method guided by multimodal semantic graphs as described in claim 1, characterized in that, The cross-modal alignment features are input into the meta-learner along with the task objective to generate task-adaptive features, including: Encode the task objective into a task objective vector; The cross-modal alignment features and the task target vector are input into the meta-learner; The meta-learner extracts common knowledge representations across different tasks; Identify the differences between the current task and historical tasks; Obtain the initial parameter matrix of the meta-learner; Based on the common knowledge representation, the spatial distribution characteristics of the parameters are determined, and key feature dimensions are extracted from the differential characteristics; Based on the parameter space distribution characteristics and key feature dimensions, the initial parameter matrix is ​​optimized by the gradient descent module to generate a transferable model parameter matrix. The feature transformation matrix is ​​adjusted using the transferable model parameter matrix to generate the adjusted feature transformation matrix. The cross-modal alignment features are processed by the adjusted feature transformation matrix to generate task-adaptive features.

7. The decision-making method guided by multimodal semantic graphs as described in claim 1, characterized in that, The task-adaptive features are input into a parallel inference network, where visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks, and cross-modal integrated reasoning sub-tasks are executed in parallel. This processes the feature content corresponding to each sub-task within the task-adaptive features, generating the output results for each sub-task, including: Assign the feature dimensions related to visual objects from the task adaptability features to the visual reasoning subtask; Assign the feature dimensions related to semantic parsing from the task-adaptive features to the language reasoning subtask; Assign the feature dimensions related to action trajectory from the task adaptability features to the action reasoning subtask; Assign the feature dimensions related to multimodal decision-making from the aforementioned task adaptability features to the cross-modal integrated reasoning subtask; The visual reasoning subtask, language reasoning subtask, action reasoning subtask, and cross-modal integrated reasoning subtask are executed synchronously in a parallel processing environment. The visual reasoning subtask processes the feature dimensions related to the visual object to generate a visual reasoning result. The language reasoning subtask processes the feature dimensions related to semantic parsing to generate language reasoning results; The action reasoning subtask processes the feature dimensions related to the action trajectory to generate action reasoning results; The cross-modal integrated reasoning subtask processes the feature dimensions related to multimodal decision-making to generate cross-modal integrated reasoning results.

8. A decision-making device guided by a multimodal semantic graph, characterized in that, The multimodal semantic graph-guided decision-making device includes: The multimodal feature extraction module is used to acquire visual data, text data, and action data, extract visual feature vectors, language feature vectors, and action feature vectors from the visual data, text data, and action data respectively, and concatenate the visual feature vectors, language feature vectors, and action feature vectors to generate multimodal initial features; The semantic mapping module is used to map the visual feature vector, language feature vector and action feature vector to the shared semantic space respectively, and generate shared semantic space features; A semantic graph construction module is used to construct a multimodal semantic graph based on the multimodal initial features; The graph-guided attention module is used to input the multimodal initial features and the multimodal semantic graph into the graph-guided attention mechanism to generate cross-modal aligned features; The task adaptation module is used to combine the cross-modal alignment features with the task target input meta-learner to generate task-adaptive features; The parallel reasoning module is used to input the task-adaptive features into the parallel reasoning network, and execute visual reasoning sub-tasks, language reasoning sub-tasks, action reasoning sub-tasks and cross-modal integrated reasoning sub-tasks in parallel in the parallel reasoning network, so as to process the feature content corresponding to each sub-task in the task-adaptive features and generate the output results of each sub-task. The decision fusion module is used to integrate the output results of each subtask through a gated fusion network to generate a global decision.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multimodal semantic graph-guided decision-making program stored in the memory and executable on the processor. When executed by the processor, the multimodal semantic graph-guided decision-making program implements the steps of the multimodal semantic graph-guided decision-making method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a decision-making program guided by a multimodal semantic graph, which, when executed by a processor, implements the steps of the decision-making method guided by a multimodal semantic graph as described in any one of claims 1-7.

Images