Multi-modal data security sharing method and system

Abstract

The invention provides a multi-modal data security sharing method and system, and relates to the technical field of computer science, and the method comprises the steps: obtaining a traditional database; constructing a database agent according to a traditional database; the method comprises the following steps: constructing a multi-modal metadata semantic space through a database agent; the method comprises the following steps: screening out participants of an adaptive task through a multi-modal metadata semantic space; performing cross-modal retrieval on the participants through an AI ready-oriented multi-modal analysis query language query method to obtain a retrieval result; locally constructing a neural network model at a participant through an adaptive machine learning modeling mechanism; according to the domain knowledge base and the knowledge graph, updating hyper-parameters and retrieval results of the neural network model; performing credible convergence on the updated neural network model through a credible execution environment module; and based on a credible convergence result, sharing the multi-modal data to each participant through a database agent.

Description

Technical Field

[0001] This invention relates to the field of computer science and technology, and in particular to a method and system for secure sharing of multimodal data. Background Technology

[0002] As a core production factor, data and databases form the core foundation for data storage and management. Their performance directly impacts data scheduling and transaction processing efficiency, and determines the overall effectiveness of complex computing tasks. With the rapid development of fields such as new materials research and development, healthcare, and finance, multimodal data, including text, images, spectra, and tables, is experiencing explosive growth. The demand for cross-organizational collaborative computing is becoming increasingly urgent. However, the heterogeneous nature of multimodal data and the complexity of cross-institutional collaboration make secure data sharing and efficient collaboration key issues that the industry urgently needs to address.

[0003] Existing database systems primarily focus on basic storage and retrieval, mainly addressing the querying and transaction management of structured or semi-structured data. While some research has introduced AI modules to enhance intelligent retrieval and data management capabilities, such optimizations are often limited to localized scenarios. Data storage employs traditional row-list expression models, query languages ​​are unimodal-oriented, and data cleaning and feature extraction rely heavily on manual operations or static rules. Model libraries are mostly designed for unimodal tasks, structural optimization depends on human experience or fixed search spaces, and independent modeling and evaluation methods are used in cross-organizational collaborative scenarios.

[0004] However, existing databases lack systematic theoretical support and cannot achieve adaptive and self-organizing full-process collaborative computing in the form of "intelligent agents". They can only passively adapt to task requirements and are unable to cope with complex scenarios. At the same time, multimodal data is stored in a scattered manner and the standards are not uniform. Differences in cross-institutional naming systems and timeline management lead to semantic fragmentation, which not only makes data retrieval and association difficult, but also lacks a dynamic participant selection mechanism that adapts to the collaborative goals. Summary of the Invention

[0005] To address the challenges of databases lacking systematic theoretical support, failing to achieve adaptive and self-organizing full-process collaborative computing in the form of "intelligent agents," and only being able to passively adapt to task requirements, making it difficult to cope with complex scenarios; and the dispersed storage and inconsistent standards of multimodal data, with differences in cross-institutional naming systems and timeline management leading to semantic fragmentation, causing difficulties in data retrieval and association, and lacking a dynamic participant selection mechanism adapted to collaborative goals, this invention provides a method and system for secure sharing of multimodal data.

[0006] The technical solutions provided by the embodiments of the present invention are as follows: A first aspect of this invention provides a method for secure multimodal data sharing, comprising: S1: Obtain a traditional database.

[0007] S2: Construct a database intelligent agent based on a traditional database.

[0008] S3: Construct a multimodal metadata semantic space through database intelligent agents.

[0009] S4: Filter out the participants for the adapted task through the multimodal metadata semantic space.

[0010] S5: Using a query method based on an AI-ready multimodal analysis query language, cross-modal retrieval of participants is performed to obtain retrieval results.

[0011] S6: Build neural network models locally on the participants through an adaptive machine learning modeling mechanism.

[0012] S7: Update the hyperparameters of the neural network model and the retrieval results based on the domain knowledge base and knowledge graph.

[0013] S8: Trusted convergence of the updated neural network model through the Trusted Execution Environment module.

[0014] S9: Based on the results of trusted convergence, multimodal data is shared with all participants through database agents.

[0015] A second aspect of the present invention provides a multimodal data secure sharing system, comprising: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the multimodal data secure sharing method as described in the first aspect.

[0016] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this invention, a database agent is constructed based on a traditional database. This agent enables adaptive and self-organizing full-process collaborative computing, efficiently handling complex scenarios. By filtering participants in the multimodal metadata semantic space, suitable participants for the task can be selected. This not only solves the semantic fragmentation problem caused by the scattered storage and inconsistent standards of multimodal data, as well as the differences in cross-organizational naming systems and timeline management, but also enables the precise selection of dynamic participants that are suitable for the collaborative goals. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a multimodal data secure sharing method provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of a multimodal data security sharing system provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0021] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0022] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0023] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0024] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0025] Reference manual attached Figure 1 The diagram shows a flowchart of a multimodal data secure sharing method provided by an embodiment of the present invention.

[0026] This invention provides a method for secure multimodal data sharing, which can be implemented by a multimodal data security sharing device, which can be a terminal or a server. The processing flow of the multimodal data security sharing method may include the following steps: S1: Obtain a traditional database.

[0027] S2: Construct a database intelligent agent based on a traditional database.

[0028] Optionally, the database intelligent agent specifically includes: an interface layer, a domain knowledge layer, a storage layer, a functional layer, a blockchain engine, an AI engine, a storage engine, a retrieval engine, and a trusted execution environment engine.

[0029] The interface layer serves as both the "task entry point and external bus." Users submit retrieval, preprocessing, and modeling requests in natural language. The Large Language Model (LLM) parses the intent into an AI-ready Multimodal Data Query Language (AQL-MD) query and an executable plan (including tasks, data scope, and dependencies). The plan is then published as an on-chain transaction via the blockchain engine, completing participant identity / quota verification and task routing, and returning tokens to the lower layers for execution. This integrates external sharing, authorization, auditing, and task orchestration (the platform's existing "blockchain and database" system and "on-chain evidence storage, off-chain data" design provide a trusted foundation for this model). The interface layer thus lowers the barrier to entry for cross-institutional access and brings governance strategies forward to the plan generation stage, ensuring that subsequent layers operate in a closed loop according to tokens and policies.

[0030] The domain knowledge layer serves as a bridge between semantics and execution, aggregating specialized knowledge bases and knowledge graphs for materials, medicine, and industry, unifying multimodal elements such as text, numerical data, and images into a shared semantic space (LLM handles text semantic embedding, Variational Autoencoder / Principal Component Analysis (VAE / PCA) processes numerical dimensionality reduction, and Contrastive Language-Image Pre-trained Model (CLIP) performs text-image alignment). It organizes entities and relationships using semantic graphs, providing constraints and evidence for query rewriting, data alignment, and task decomposition. Simultaneously, this layer outputs computable metrics such as quality and freshness, supporting participant selection and alliance building (Graph Neural Networks (GNNs) learn the task-data-participant ternary relationship, and Deep Reinforcement Learning (DRL) performs dynamic selection for multi-objective optimization). These knowledge assets are managed with versions and evidence chains for sharing and auditing by the interface and functional layers.

[0031] The functional layer serves as the execution hub, using an AQL-MD parser / optimizer to compile high-level semantic plans into cost-aware execution DAGs. The retrieval end employs a hybrid inverted / vector / graph index with learned rearrangement. The data preparation end follows an "AI-Ready" paradigm, using DAGs and reinforcement learning to search for optimal routes for operators such as anomaly handling, missing data completion, and feature construction, producing plug-and-play datasets. The modeling end utilizes AdaptML for Neural Architecture Search (NAS) and hyperparameter co-optimization, collaborating with federated learning / multi-party secure computation in cross-organizational scenarios, and combining TEE's secure aggregation to achieve "usable but invisible" joint training and deployment. End-to-end metrics, model / data fingerprints, and access logs are written back to the blockchain, ensuring traceability and auditability of results.

[0032] The storage layer uses a Dynamic Container Model (DCM) to uniformly host multimodal raw data, feature vector tables, and metadata graphs. The DCM decouples the schema from the value through a two-layer mechanism of "templates and instances," adapting to complex structures such as variable hierarchies, nesting, and generators. Instances strictly adhere to template mapping, supporting cross-modal expansion while maintaining version and lineage consistency. The underlying layer interfaces with relational, object, and document / time-series libraries, establishing a multi-dimensional index combining columnar, vectorized, and graph approaches to support high-concurrency data supply. The governance side employs an "on-chain policy / log / fingerprint, off-chain raw data" model to achieve tamper-proof traceability, forming a closed loop with the AI-ready retrieval and preprocessing of the upper-layer functions.

[0033] In this embodiment of the invention, a database intelligent agent is constructed based on a traditional database. This fully reuses the mature storage architecture, data management mechanism, and existing data assets of the traditional database. Intelligent capabilities can be injected without overturning the original system. This reduces the cost and risk of technology migration and reconstruction. The intelligent agent can also realize functions that traditional databases lack, such as automatic optimization of query plans, dynamic load adaptation, and multimodal data collaborative processing. At the same time, it meets the needs of cross-organizational collaborative scenarios, upgrading the original database from "passive storage and computing" to "proactive intelligent decision-making". This provides efficient, compatible, and reliable underlying data intelligence support for subsequent core processes such as adaptive modeling and multi-objective optimization.

[0034] S3: Construct a multimodal metadata semantic space through database intelligent agents.

[0035] Among them, the multimodal metadata semantic space is based on multimodal representation and domain ontology. It maps metadata of different types of data such as text, images, spectrograms, and tables into comparable vectors and relational representations, realizing the semantic alignment and unified cataloging of heterogeneous metadata. At the same time, it incorporates quality assessment indicators such as data availability, compliance, and timeliness, providing semantic support for data retrieval and matching, dynamic participant screening, and collaborative modeling in cross-organizational scenarios, breaking down semantic barriers between different sources and types of data.

[0036] In one possible implementation, S3 specifically includes: S301: Obtain raw data.

[0037] The raw data consists of raw text, numerical and image data from the databases, object storage and file systems of the participating units, as well as basic descriptive information such as source, collection time, version number, authorization and permissions. These are defined as the raw text set RawText, the raw numerical set RawNum, the raw image set RawImg, and the standardized pre-metadata RawMeta.

[0038] S302: Extract raw data through a database agent.

[0039] It should be noted that the database agent (DB-Agent) uses an access adapter to extract the original text set RawText, the original numerical set RawNum, the original image set RawImg, and the standardized pre-metadata RawMeta, and records the data source fingerprint and access log to form a verifiable initial chain of evidence.

[0040] S303: Standardize the raw data to obtain standardized data.

[0041] It should be noted that on the text side, sentence segmentation, noise reduction, stop word removal, and terminology standardization are performed to obtain the cleaned text CleanText; on the numerical side, missing value imputation, anomaly detection, and unit unification are performed to form a standardized numerical set NumStd; on the image side, resolution / channel / cropping unification and noise reduction enhancement are performed to obtain a standardized image set ImgStd; and on the metadata side, field mapping, format validation, and permission tag completion are completed to obtain standard metadata MetaStd.

[0042] Furthermore, standardizing the raw data can ensure the comparability and alignment of data from different sources at the semantic and format levels, providing consistent input for representation learning.

[0043] S304: Perform unified representation learning and cross-modal alignment on standardized data to obtain cross-modal data.

[0044] It should be noted that the use of large language models to obtain high-dimensional semantic representations of text is defined as text embedding (Emb). text And obtain text cluster labels (Cluster) through clustering. text This is used as a representative semantic for subsequent image-text alignment; for numerical data, principal component analysis is first used to obtain a linear low-dimensional mapping, followed by variational autoencoder to learn nonlinear latent variables, and finally fusion to obtain the numerical semantic embedding (Emb). num It also calculates a consistency metric on the text side, defined as a numerical-text alignment matrix. Using contrastive language, the image pre-trained model performs cross-modal alignment between the image and text representing semantics to obtain the image semantic embedding (Emb). img Image and text alignment matrix By using the three types of embedding and two types of alignment described above, text, numerical values, and images can be compared and computed in a unified semantic space.

[0045] S305: Based on cross-modal data, calculate the similarity and consistency between modalities to form a cross-modal semantic relevance matrix.

[0046] S306: Construct a metadata weight vector based on standard metadata.

[0047] S307: Generate a metadata semantic graph based on the cross-modal semantic relevance matrix and metadata weight vector.

[0048] The metadata semantic graph MetaGraph uses "tasks, data, and participants" as nodes and "semantic associations, dependencies, and historical collaborations" as weighted edges.

[0049] S308: Map the cross-modal semantic relevance matrix and metadata weight vector to edge weights and time attributes, establish a hybrid index set, and obtain a multimodal metadata semantic space. The multimodal metadata semantic space specifically includes the cross-modal semantic relevance matrix, metadata weight vector, metadata semantic graph, and hybrid index set.

[0050] The IndexMix hybrid index set includes inverted indexes oriented towards keywords and structure fields, vector indexes oriented towards embedded vectors, and graph indexes oriented towards path queries.

[0051] In this embodiment of the invention, by standardizing processing, unifying cross-modal representation and semantic alignment, the semantic and format barriers of heterogeneous data such as text, images and spectrograms are broken down. At the same time, relying on database intelligent agents to ensure data traceability and the integrity of the evidence chain, and using metadata semantic graphs and hybrid indexes to achieve semantic cataloging and efficient retrieval of data, it not only provides accurate semantic support for data matching and participant selection in cross-organizational scenarios, but also ensures the reliability and adaptability of collaborative data by incorporating quality assessment indicators and weight vectors, laying a unified, reliable and computable semantic foundation for subsequent adaptive modeling, multi-objective optimization and other processes.

[0052] S4: Filter out the participants for the adapted task through the multimodal metadata semantic space.

[0053] In one possible implementation, S4 specifically includes: S401: Input the metadata semantic graph and hybrid index set of the multimodal metadata semantic space into the representation learning and decision optimization module.

[0054] S402: Through graph neural networks, the nodes of the metadata semantic graph are iteratively aggregated to obtain the node embedding set and the matching score of the task participants.

[0055] Graph neural networks (GNNs) are primarily used to process the graph structure information of multimodal data and are one of the core technologies for semantic alignment and relationship modeling. They can learn features from the graph structure in the semantic space of multimodal metadata, capturing deep semantic relationships between different modalities and cross-organizational participants by aggregating the attributes of nodes themselves and the association information of neighboring nodes. This optimizes the matching accuracy of multimodal data retrieval, the rationality of dynamic participant selection, and the feature flow and dependencies between nodes in adaptive model construction, providing intelligent analysis capabilities at the graph structure level for secure sharing and efficient collaborative computing of cross-organizational multimodal data.

[0056] Specifically, a graph neural network is used to iteratively aggregate the "task, data, and participant" nodes of the metadata semantic graph MetaGraph to obtain the node embedding set NodeEmb. And calculate the MatchScore(Task) of the task participants. Party).

[0057] S403: Determine the constraint profile of the participants based on standard metadata.

[0058] Specifically, information such as resource capacity, privacy and compliance constraints, and network latency are derived from the standard metadata MetaStd to form a constraint profile of the participants.

[0059] S404: Based on the matching scores and constraint profiles of the task participants, output the optimal collaborative computing alliance and execution strategy through deep reinforcement learning.

[0060] Deep reinforcement learning is an important branch of machine learning that combines the perceptual capabilities of deep learning with the decision-making abilities of reinforcement learning. It learns optimal policies through continuous interaction between the agent and its environment. Its core mechanism involves the agent receiving reward signals from the environment after performing an action, and then using deep neural networks to approximate value functions or policy functions to handle complex decision-making problems in high-dimensional state spaces.

[0061] Specifically, the MatchScore (Task) is the matching score of the task participants. The algorithm uses the Party and the constrained profiles of the participants as states, and employs deep reinforcement learning to optimize the collaboration among multiple objectives such as coverage, semantic similarity, reputation, and constraint satisfaction, outputting the optimal collaborative computing alliance (denoted as Alliance). (clearly indicating members and their roles) and their execution strategy (denoted as Policy) alliance (Explicitly specify routing, parallelism, and quotas).

[0062] S405: Select suitable participants for the task based on the optimal collaborative computing alliance and execution strategy.

[0063] Furthermore, based on the optimal collaborative computing alliance With Execution Policy alliance The task-consortium mapping is registered in the MetaGraph and the IndexMix, generating searchable task matching entries (MatchEntries) for subsequent reuse and auditing. During the actual collaborative execution phase, operational data such as success rate, end-to-end latency, privacy budget consumption, and performance metrics are collected and stored as a FeedbackLog. This FeedbackLog serves as an online update signal fed back to consortium generation and strategy optimization, further updating the node embedding set (NodeEmb). With optimization strategies; when significant shifts in semantics or distribution are detected, they can also be used as trigger conditions to backflow to unified cross-modal embedding and alignment or semantic graph construction and hybrid indexing, and perform embedding calibration and incremental updates of semantic graphs or indexes.

[0064] In this embodiment of the invention, relying on the deep semantic support of the multimodal metadata semantic space, the semantic association between tasks and participants is accurately captured through graph neural networks. At the same time, a participant constraint profile is generated by combining standard metadata. Then, deep reinforcement learning is used to achieve a trade-off optimization of multiple objectives such as coverage, semantic similarity, and compliance. This not only selects the optimal collaborative computing alliance with strong adaptability, resource matching, and compliance with constraints, but also realizes the dynamic iteration of alliances and strategies through the running data feedback and return mechanism. Furthermore, the reusability and auditability of the selection results can be ensured through task matching item registration. This effectively solves the problems of accuracy, compliance, and dynamic adaptation in participant selection in cross-organizational scenarios, and provides core organizational guarantees for the efficient and secure development of subsequent collaborative computing.

[0065] S5: Using a query method based on an AI-ready multimodal analysis query language, cross-modal retrieval of participants is performed to obtain retrieval results.

[0066] Among them, the AI-ready multimodal analysis query language relies on the Large Language Model (LLM) proxy framework and multimodal metadata semantic space. It supports users to input complex query intents in natural language, and automatically decomposes them into sub-tasks such as structured data filtering, image analysis, and feature extraction, and plans the optimal execution process to achieve unified retrieval of multimodal data such as text, images, and spectrograms. At the same time, it integrates an automated preprocessing module, which can complete data cleaning, heterogeneous feature alignment, and standardized extraction for AI modeling needs, and output directly usable modeling input data. It can also ensure the interpretability of the query process through task link tracing and intermediate result visualization, providing efficient and accurate language layer support for intelligent retrieval, dynamic participant selection, and adaptive modeling in cross-organizational collaborative scenarios.

[0067] In one possible implementation, S5 specifically includes: S501: Construct a dynamic container model, which specifically includes: templates, instances, and a dynamic container tree.

[0068] The dynamic container model adopts a two-layer, one-tree design of "template set - instance set - dynamic container tree". The template set defines the data structuring rules and types, the instance carries specific multimodal data such as text, image, and spectrogram, and the dynamic container tree connects the two through attribute nodes, hierarchical relationships and type mapping. It breaks the storage limitations of fixed rows and columns in traditional databases, realizes the unified collection, version control and semantic association of heterogeneous data, and provides structured support for multimodal metadata semantic alignment, AI-ready query optimization and cross-organizational collaborative computing.

[0069] It should be noted that the Dynamic Container Model (DCM) is built with a "two-layer, one-tree" structure: the first is the template set M, and the second is the instance set I; the two are linked by a dynamic container tree DC-Tree with type annotations.

[0070] The specific expression for the dynamic container tree is as follows: ; in, T DC Represents a dynamic container tree. V Represents a set of attribute nodes. E Indicates a hierarchical or complex relationship. τ Represents type mapping.

[0071] S502: Map multimodal data to fields of a template to form an instance.

[0072] It should be noted that a version number Ver and a fingerprint digest Hash are generated for each instance.

[0073] S503: Assign registration numbers to the template and instance respectively, and obtain the template identifier and instance identifier.

[0074] It should be noted that a registration number, TemplateID, and an instance ID are assigned to the template and instance respectively, and the entry log is recorded to ensure traceability.

[0075] S504: Based on the template identifier, instance identifier, and dynamic container tree, generate the path directory, shard set, and multimodal auxiliary index through the path navigation function.

[0076] The path navigation function serves as a mapping bridge between user queries and multimodal data storage structures. It receives path expressions from AQL-MD, parses and maps them to the attribute node exponent set of the dynamic container tree, thereby accurately locating the storage location of multimodal data such as text, images, and spectrograms in the dynamic container model. This provides a foundation for decomposing query statements into semantically independent subtasks, constructing dependency graphs, and subsequent cost-aware parallel optimization, ensuring the accuracy and efficiency of cross-modal data retrieval.

[0077] Among them, path navigation function π The specific expression is: ; in, Paths Represents the path expression for AQL-MD, 2 V This represents the power set of DC-Tree nodes.

[0078] S505: Using a large language model, natural language queries are parsed into multimodal analysis query language statements.

[0079] Among them, the large language model is a large-scale language processing system built on deep learning technology. With the Transformer architecture as its core, it masters language syntax, semantics and contextual logic through self-supervised pre-training on massive amounts of unlabeled text data, and then fine-tunes it to adapt to specific scenarios. Its self-attention mechanism can efficiently capture long-distance relationships in text. After the parameter scale reaches a critical value, it will also exhibit emergent capabilities such as context learning, instruction following, and logical reasoning. It can realize diverse tasks such as text generation, multilingual translation, code development, and intelligent question answering, and is widely used in content creation, customer service, education, medical care, finance and other fields.

[0080] S506: Decompose multimodal analysis query language statements into semantically independent subtasks through path navigation functions.

[0081] S507: Construct a dependency graph with semantically independent subtasks as nodes and data dependencies as edges; and extract the cost features of semantically independent subtasks.

[0082] S508: Based on semantically independent subtasks, dependency graphs, and cost characteristics, a job description is generated through a self-optimizing data processing plan.

[0083] The dependency graph is constructed based on semantically independent subtasks (such as multimodal data retrieval, cleaning, and feature extraction) after path navigation function parsing. Each node corresponds to a single subtask, and directed edges represent data dependencies or execution order constraints between subtasks (such as needing to complete structured data screening before performing unstructured image feature extraction). It can clearly sort out the logical connections and sequence of multimodal data processing, providing a structured basis for subsequent cost-aware parallel optimization and scheduling strategy formulation, ensuring that the cross-modal data processing process is orderly and efficient, and avoiding task conflicts and redundant calculations.

[0084] In one possible implementation, S508 specifically includes: S5081: Based on semantically independent subtasks, dependency graphs, and cost characteristics, a parallel execution plan and scheduling strategy are obtained through a cost model optimizer.

[0085] Among them, the cost model optimizer is the core component of modern database query optimization. Based on statistical information such as table data volume, column distribution, and index status, it quantifies the execution costs of IO (data reading) and CPU (data processing) through mathematical models, generates multiple candidate query plans (such as different join orders and index selection), evaluates their total cost, and finally selects the plan with the lowest resource consumption for execution. Compared with traditional rule-based optimizers, it is more adaptable to complex queries and changes in data distribution, and is a key technology for improving database query efficiency.

[0086] S5082: The cleaning process is formalized as a directed acyclic graph by cleaning the primitive library and quality metric function across modalities.

[0087] Among them, the quality metric function is the core tool for ensuring the quality of multimodal data. It is mainly applied to the semantic space of multimodal metadata. By quantifying key indicators such as data availability, compliance, timeliness, and accuracy, it objectively evaluates the quality of heterogeneous data such as text, images, and spectrograms. It provides a quantitative basis for data screening, dynamic participant matching, and AI modeling input data verification in cross-organizational collaborative computing, ensuring that the data quality used for retrieval matching and collaborative modeling meets the standards, and avoiding low-quality data from affecting the collaborative computing effect and model performance.

[0088] Directed acyclic graphs (DAGs) are the core structure supporting multimodal data processing and adaptive modeling. They consist of nodes (V) representing specific operations or tasks and directed edges (E) defining the execution order / data flow. A key characteristic is the absence of cyclic dependencies. DAGs are used to construct multimodal data processing chains (such as decomposing queries into ordered subtasks like retrieval, cleaning, and feature extraction) and also serve as the search space framework for adaptive modeling (integrating operational units such as convolution and attention). Through clear logical connections and the absence of cyclic constraints, they ensure orderly and efficient data processing flows and traceable model construction, providing structured support for task scheduling and parallel optimization in cross-organizational collaborative computing.

[0089] S5083: Based on the directed acyclic graph, reinforcement learning is used to model the cleaning and orchestration as a Markov decision process, resulting in the optimal directed acyclic graph for cleaning, the parameter set, and the cleaning plan.

[0090] Reinforcement learning is a machine learning method based on "trial and error feedback." Its core principle is to allow an agent to continuously interact with its environment, adjusting its strategy based on reward signals (positive / negative feedback) generated by its actions, and gradually learning the optimal course of action to maximize cumulative rewards. It includes the agent (the subject performing the action), the environment (the interaction scenario), the action (the agent's behavioral choices), the state (the current state of the environment), and the reward (feedback signal). It does not require pre-provided labeled data but adapts to complex dynamic scenarios through a dynamic balance of exploration and utilization.

[0091] Markov decision processes (MDFs) are a core mathematical framework for reinforcement learning. With the "Markov property" (the current state depends only on the previous state and the actions performed, and is independent of history) as its core characteristic, MDFs accurately model the interactive decision-making problems between agents and dynamic environments through five key elements: state space, action space, state transition probabilities, reward function, and discount factor. They can derive optimal decision strategies based solely on the current state without relying on complete historical information. In scenarios such as multimodal data processing and database optimization, they can be used to model dynamic decision-making scenarios such as query execution plan scheduling, multimodal task allocation, and adaptive modeling parameter adjustment, providing theoretical support for reinforcement learning algorithms and helping systems find the decision scheme with the best long-term returns.

[0092] S5084: Assemble the optimal cleaning directed acyclic graph, parameter set, cleaning plan, parallel execution plan and scheduling strategy into a job description.

[0093] S509: According to the job description, execute the AI-ready output and loading specifications to obtain the search results.

[0094] Furthermore, based on the JobSpec, parallel retrieval and cleaning orchestration are performed in a distributed environment to batch process the parsed data; the process automatically generates lineage and processing logs.prep (Operator, parameters, time, version, path) and record caching or failure retry information. Output an AI-Ready dataset that can be directly consumed for training or inference. AR (Packaged in the standard format for modality / task), and generating a LoaderSpec (segmentation, batching, feature columns, label mapping, random seed, normalization / vocabulary / pixel pipeline, etc.) and an optional FeatureStore. Also included are the LoaderSpec, FeatureStore, Catalog directory, and multimodal auxiliary index Idx. aux Create a reusable CatalogEntry entry for easy reuse in subsequent queries and training.

[0095] Furthermore, execution-period quality and efficiency metrics are collected to form a feedback log (throughput, latency, cache hits, ΔQ changes, downstream metric gains, anomalies, and retry records). The feature storage snapshot Feedback Log is used as an online signal: on the one hand, it updates the reinforcement learning strategy and Param... This involves solidifying high-yield paths and eliminating inefficient primitive combinations; on the other hand, it involves adjusting Catalog and Idx based on hot topics and failure patterns. aux And ShardSet performs incremental maintenance (new indexes, resharding, cache preheating), and triggers template evolution when necessary.

[0096] In this embodiment of the invention, relying on an AI-ready multimodal analysis query language and combining a large language model proxy framework, users can efficiently initiate cross-modal queries using natural language. A dynamic container model is used to achieve unified aggregation and accurate positioning of heterogeneous data. Then, a self-optimizing data processing plan (integrating cost model, reinforcement learning, and directed acyclic graph) completes subtask decomposition, parallel scheduling, and high-quality cleaning. This outputs ready data that is directly adapted to AI modeling, and ensures query interpretability through link tracing and visualization. At the same time, it meets the requirements of convenience, accuracy, and efficiency of multimodal data retrieval in cross-organizational scenarios, greatly reducing the threshold for data use and providing high-quality, reusable core data input for subsequent collaborative modeling.

[0097] S6: Build neural network models locally on the participants through an adaptive machine learning modeling mechanism.

[0098] Among them, the adaptive machine learning modeling mechanism is the core modeling framework that adapts to multimodal heterogeneous data and dynamic collaborative scenarios. It uses a directed acyclic graph (DAG) as the model search space carrier, integrates the dynamic decision-making capabilities of reinforcement learning and Markov decision processes (MDP), and combines real-time data quality feedback from quality metric functions. It can automatically optimize the model structure (such as adding or removing network layers, adjusting the combination of operation units), hyperparameter configuration, and data processing links according to the distribution changes of multimodal data such as text, images, and spectrograms, and the needs of cross-organizational collaborative tasks (such as modeling accuracy and computational efficiency priority). It can adapt to dynamic scenarios without manual intervention, while ensuring the accuracy and stability of modeling results, and providing efficient and flexible intelligent modeling support for cross-organizational multimodal data collaborative computing.

[0099] In one possible implementation, S6 specifically includes: S601: Construct a candidate neural network model library based on the directed acyclic graph.

[0100] S602: Construct a search space in the candidate neural network model library using reusable functional blocks as the smallest unit.

[0101] The expression for the search space is as follows: ; Where G represents the search space, V Represents the set of candidate operation nodes. E Directed edges representing the characteristic flow direction.

[0102] It should be noted that by introducing a library of reusable functional blocks and establishing a mapping table from modalities to reusable functional blocks, the library of reusable functional blocks or the mapping table of reusable functional blocks is injected into the search space, resulting in a modular search space.

[0103] S603: Select a set of candidate subnets based on the search space using a controller that employs reinforcement learning or evolutionary search.

[0104] S604: Build a policy controller.

[0105] Among them, the policy controller is the core decision-making and execution component for multimodal data collaborative computing and adaptive modeling. It integrates the dynamic decision-making logic of reinforcement learning and Markov decision processes (MDPs), as well as the feedback data of quality metric functions and cost model optimizers. Around the core objectives of cross-organizational multimodal data processing (such as modeling accuracy, query efficiency, and compliance), it dynamically formulates and executes the optimal strategy. At the same time, it responds in real time to dynamic scenarios such as changes in data distribution and adjustments to task requirements, ensuring that the entire collaborative computing system achieves a balance between efficient operation, quality compliance, and compliance and security, and provides stable decision support for end-to-end multimodal data processing and intelligent modeling.

[0106] S605: Evaluate the candidate subnet set using the multi-objective reward function and constraint set in reinforcement learning.

[0107] The multi-objective reward function is the core feedback mechanism of reinforcement learning and policy controllers. Its core function is to balance multiple conflicting or complementary objectives in multimodal collaborative computing, avoiding system imbalance caused by optimizing a single objective. By quantifying the achievement of key objectives such as modeling accuracy, query efficiency, data compliance, resource utilization, and cross-organizational task collaboration, and assigning objective weights based on scenario requirements (e.g., emphasizing accuracy rewards in high-precision modeling scenarios and increasing efficiency weights in resource-constrained scenarios), multi-dimensional objectives are transformed into a unified reward signal fed back to the policy controller, guiding the reinforcement learning algorithm to optimize decision-making strategies.

[0108] S606: Based on the evaluation results, update the strategy controller to obtain the polymer network set and the evaluation set.

[0109] S607: Assemble candidate fusion models according to the fusion topology based on the polymer network set.

[0110] Among them, the fusion topology is the core structural framework of multimodal data collaborative computing and adaptive modeling. Based on the topological structure, it integrates core components such as multimodal metadata semantic space, directed acyclic graph (DAG), and dynamic container model. It abstracts multimodal data modules, modeling operation units, and cross-organizational participants into topological nodes. By defining the data flow, task collaboration, and feedback interaction relationships (topological edges) between nodes, it breaks down the collaboration barriers of heterogeneous data, scattered modules, and cross-organizational scenarios, and realizes end-to-end linkage of data processing, model building, and task scheduling. It provides structured support for the decision execution of policy controllers, the interactive feedback of reinforcement learning, and multi-objective optimization, ensuring the orderliness and efficiency of the entire collaborative computing system.

[0111] S608: Introduces a shared parameter supernetwork and a weighted shared index to perform hot start and lightweight fine-tuning of candidate fusion models, obtain evaluation results, and register the evaluation results to the subnet storage pool and performance cache.

[0112] Among them, the shared parameter hypernetwork is the core network architecture of the adaptive machine learning modeling mechanism. With parameter sharing as its core design, it integrates multimodal feature extraction, model structure search and cross-scene adaptation capabilities. Its hypernetwork framework contains multiple sub-network modules that adapt to different modalities (text, image, spectrogram). Core parameters are reused among sub-modules to reduce redundant calculations. At the same time, it dynamically selects sub-network combinations and parameter configurations based on the directed acyclic graph (DAG) search space. It can efficiently integrate multimodal heterogeneous features and adapt to changes in data distribution and task requirements in cross-organizational collaborative scenarios. It provides lightweight network support for the model structure and hyperparameter adaptive optimization driven by the policy controller, ensuring a balance between modeling efficiency and result accuracy.

[0113] Among them, the weight sharing index is the core index structure that supports the efficient operation of shared parameter hypernetworks. In response to the parameter sharing requirements of multimodal sub-modules of hypernetworks, the shared core parameters of each modality (text, image, spectrogram) sub-module are structured and organized. By associating the modality type, model structure position, adapted task scenario and reuse priority of the parameters, an efficient index mapping relationship is constructed. This can not only realize the rapid location and call of shared parameters, reduce the overhead of repeated parameter storage and search, but also support the accurate matching of parameters when the policy controller dynamically adjusts the model structure. It ensures the efficiency of parameter reuse and the stability of multimodal feature fusion in the adaptive modeling process, and provides parameter-level index support for the lightweight and efficient multimodal collaborative computing across organizations.

[0114] S609: Based on candidate fusion models, weight sharing indexes, subnet storage pools, and performance caches, a neural network model is obtained by aggregating gradients across multiple subnets.

[0115] Among them, multi-subnetwork gradient aggregation is the core training optimization mechanism of shared parameter supernetwork. It addresses the gradient dispersion problem during the training of multimodal subnetworks (text, image, spectrogram, and other adapted subnetworks) under the supernetwork. By collecting the local gradients of each subnetwork in its own modal data training, and combining modality importance, task priority, and data quality weights, it performs weighted integration to generate a unified global gradient and backpropagates it, thereby achieving collaborative updates of shared core parameters.

[0116] In one possible implementation, S609 specifically includes: S6091: Based on the evaluation results, the weights of the shared operations are updated by aggregating gradients across multiple subnets. ; ; in, This represents the gradient after aggregation. ω s This represents the gradient weight factor of the subnet. This represents the loss gradient of the subnet. W op The weight parameter represents the weight of the sharing operation. η This represents the learning rate.

[0117] S6092: Combining the weights of the updated shared operations, and based on the performance cache, identify frequently used modules and high-contribution modules.

[0118] S6093: Based on the statistical results, construct a set of priority paths and update the set of priority paths.

[0119] Among them, the priority path set serves as the basis for efficient path selection in multimodal data processing and adaptive modeling. Based on the cost evaluation results of the cost model optimizer and the quality feedback of the quality metric function, a high-priority path set that balances efficiency, accuracy, and resource consumption is selected from all potential execution paths in the directed acyclic graph (DAG) (such as multimodal query processing links and subnet combination paths of shared parameter hypernetworks). This not only clarifies the optimal execution logic for multimodal data retrieval and model training but also supports rapid decision-making by the policy controller, reduces path search overhead in dynamic scenarios, and ensures the efficiency and stability of cross-organizational collaborative computing and adaptive modeling.

[0120] It should be noted that, based on the historical usage frequency and performance contribution of each module, the module weight is calculated; a priority path set is dynamically constructed according to the weight score, and each path is assigned a priority score composed of usage frequency, performance contribution, and computational efficiency; a periodic elimination mechanism is established to remove inefficient paths, and the path priority and computational resource allocation are dynamically adjusted according to real-time training feedback; by merging duplicate modules, establishing a computation result cache, and realizing cross-task reuse, redundant computation is effectively reduced, forming a continuously optimized adaptive computation path management mechanism.

[0121] S6094: Based on the updated set of priority paths, combined with knowledge distillation or regression distillation, a neural network model is obtained.

[0122] Knowledge distillation is a core technology for lightweighting and performance optimization of shared parameter hypernetworks. Its core logic is to transfer the knowledge (such as feature representation, decision logic, and probability distribution) of a mature, complex multimodal large model (teacher model) to a more streamlined student model (a lightweight subnetwork adapted for cross-organizational collaboration) through methods such as temperature coefficient adjustment and soft label supervision. This retains the multimodal feature fusion capability and modeling accuracy while significantly reducing the model parameter size, computational overhead, and storage requirements. At the same time, it improves the model inference speed, perfectly adapting to the resource-constrained and low-latency requirements of cross-organizational multimodal collaborative computing scenarios, and providing technical support for the efficient implementation of adaptive modeling.

[0123] Among them, regression distillation is a specialized implementation of knowledge distillation for regression tasks. By using regression loss functions such as mean squared error (MSE), it accurately transfers the continuous value prediction knowledge (including output results and intermediate features) of the teacher model to the lightweight student model. This avoids the distribution transformation error of classification distillation while preserving the numerical accuracy and trend correlation of the regression task. It supports the lightweighting of multimodal subnetworks of shared parameter hypernetworks, reduces computation and storage overhead while maintaining the accuracy of cross-modal regression tasks (such as numerical prediction and feature quantization), and adapts to the resource-constrained needs of cross-organizational collaboration.

[0124] It should be noted that, if deployment is required, the neural network model will be pruned, quantized, and lightly distilled according to constraints to generate the deployment model and its specifications.

[0125] Furthermore, the (structural fingerprint, shared weights, evaluation results, or online metrics) are written into the structured cache index I. cache This process generates searchable and reusable entries, and produces model cards (data source, training or distillation configuration, metrics, applicable boundaries, retraining or retirement conditions). When similar tasks or similar modal distributions occur, the system can directly retrieve data from I... cache Loading weights for hot start-up or verification significantly shortens subsequent cycles. Simultaneously, monitoring data such as runtime accuracy, latency, and computing power consumption are collected to form a feedback log. The benefits or bottleneck characteristics in the log are used to adjust values ​​and sampling directions, as well as to update the priority of the neural network model or priority path set. If long-term observations show that the modal distribution or task objective has drifted, incremental evolution of the functional block library or mapping table can be triggered.

[0126] In this embodiment of the invention, relying on an adaptive machine learning modeling mechanism, the model is built locally by the participants, which not only ensures data security and avoids the risk of cross-organizational data transmission, but also achieves automatic optimization of model structure and hyperparameters through modular search space, policy controller and multi-objective reward function, which can adapt to the distribution of multimodal data and changes in task requirements without manual intervention. At the same time, the shared parameter hypernetwork and weight shared index reduce redundant calculations, multi-subnet aggregated gradient ensures the consistency of multimodal feature fusion, knowledge distillation / regression distillation realizes model lightweighting, priority path set and structured cache index enable the model to be quickly hot-started and reused and shorten the iteration cycle. Then, the model dynamic iteration and functional block evolution are driven by feedback logs, which not only takes into account modeling accuracy, efficiency and resource adaptability, but also provides flexible, efficient and secure local modeling support for cross-organizational collaborative scenarios, which significantly reduces the cost and threshold of collaborative modeling.

[0127] S7: Update the hyperparameters of the neural network model and the retrieval results based on the domain knowledge base and knowledge graph.

[0128] In one possible implementation, S7 specifically includes: S701: Update the search results based on the domain knowledge base and knowledge graph.

[0129] S702: Update the hyperparameters of the neural network model based on the domain prior distribution and constraints in the domain knowledge base.

[0130] In this embodiment of the invention, the professional support of domain knowledge base and knowledge graph is introduced. This not only integrates prior domain knowledge into the search results and corrects data biases, improving the scenario adaptability of the data required for cross-organizational collaboration, but also optimizes the hyperparameters of the neural network model based on the prior domain distribution and constraints, avoiding blind hyperparameter search and making the model more in line with the essence of the business scenario. At the same time, it ensures the domain rationality of the search results and model parameters, reduces invalid iterations, and provides key professional knowledge support for improving the accuracy and scenario adaptability of cross-organizational collaborative modeling.

[0131] S8: Trusted convergence of the updated neural network model through the Trusted Execution Environment module.

[0132] Among them, the Trusted Execution Environment (TEE) module is a core component for ensuring the security of cross-organizational multimodal collaborative computing. It constructs a secure execution space independent of the operating system through hardware-level isolation technology, encrypting and protecting the transmission, storage, and computation of multimodal data (text, images, spectrograms, etc.) while preventing data tampering, malicious theft, and unauthorized access. It is compatible with core processes such as model training and multi-objective optimization of shared parameter hypernetworks, and meets the compliance requirements of cross-organizational data collaboration, providing underlying security support for sensitive data processing and trusted collaboration.

[0133] In this embodiment of the invention, relying on the hardware-level isolation and encryption capabilities of the Trusted Execution Environment module, secure and reliable aggregation of updated neural network models is achieved in cross-organizational scenarios. This not only prevents model data from being tampered with, stolen, or accessed without authorization during the aggregation process, but also protects the privacy and security of local data and model details of participating parties. At the same time, it meets the compliance requirements of cross-organizational collaboration, provides an underlying security safety net for the entire multimodal collaborative modeling process, and ensures the integrity and credibility of the final aggregated model.

[0134] S9: Based on the results of trusted convergence, multimodal data is shared with all participants through database agents.

[0135] In this embodiment of the invention, relying on the mature adaptation and management capabilities of database agents, compliant and efficient sharing of multimodal data is achieved. This not only continues the security guarantee of the trusted execution environment, ensuring the integrity and privacy of shared data / models, but also avoids cross-organizational system adaptation barriers by leveraging the compatibility of database agents, allowing all participants to easily obtain collaborative results. At the same time, the shared data can feed back into the local model iteration and new task adaptation of the participants, forming a collaborative closed loop and improving the overall value and iteration efficiency of cross-organizational multimodal collaborative computing.

[0136] Reference manual attached Figure 2 The diagram shows a structural schematic of a multimodal data security sharing system provided by the present invention.

[0137] The present invention also provides a multimodal data secure sharing system 20, applied to the above-mentioned multimodal data secure sharing method, comprising: Processor 201.

[0138] The memory 202 stores computer-readable instructions that, when executed by the processor 201, implement the multimodal data secure sharing method as described in the method embodiment.

[0139] The multimodal data secure sharing system 20 provided by the present invention can execute the above-described multimodal data secure sharing method and achieve the same or similar technical effects. To avoid duplication, the present invention will not elaborate further.

[0140] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0141] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0142] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0143] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0144] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0145] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0146] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0148] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0151] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal data secure sharing method as described in the method embodiments.

[0153] The computer-readable storage medium provided by this invention can implement the steps and effects of the multimodal data secure sharing method of the above method embodiments. To avoid repetition, this invention will not repeat them.

[0154] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0155] The following points need to be explained: (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0156] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0157] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0158] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for secure sharing of multimodal data, characterized in that, include: S1: Obtain a traditional database; S2: Construct a database intelligent agent based on the traditional database; S3: Construct a multimodal metadata semantic space through the database agent; S4: Filter out the participants in the adaptation task through the multimodal metadata semantic space; S5: Using a query method based on an AI-ready multimodal analytics query language, perform cross-modal retrieval on the participants to obtain retrieval results; S6: Build neural network models locally on the participants through an adaptive machine learning modeling mechanism; S7: Update the hyperparameters of the neural network model and the retrieval results based on the domain knowledge base and knowledge graph; S8: Trusted convergence of the updated neural network model is performed through the Trusted Execution Environment module; S9: Based on the results of trusted convergence, the multimodal data is shared with each of the participating parties through the database agent.

2. The multimodal data secure sharing method according to claim 1, characterized in that, The database intelligent agent specifically includes: an interface layer, a domain knowledge layer, a storage layer, a functional layer, a blockchain engine, an AI engine, a storage engine, a retrieval engine, and a trusted execution environment engine.

3. The multimodal data secure sharing method according to claim 1, characterized in that, S3 specifically includes: S301: Obtain raw data; S302: Extract the raw data through the database agent; S303: Standardize the original data to obtain standardized data; S304: Perform unified representation learning and cross-modal alignment on the standardized data to obtain cross-modal data; S305: Based on the cross-modal data, calculate the similarity and consistency between modalities to form a cross-modal semantic relevance matrix; S306: Construct a metadata weight vector based on standard metadata; S307: Generate a metadata semantic graph based on the cross-modal semantic relevance matrix and the metadata weight vector; S308: Map the cross-modal semantic relevance matrix and the metadata weight vector to edge weights and time attributes, establish a hybrid index set, and obtain the multimodal metadata semantic space, wherein the multimodal metadata semantic space specifically includes the cross-modal semantic relevance matrix, the metadata weight vector, the metadata semantic graph, and the hybrid index set.

4. The multimodal data secure sharing method according to claim 1, characterized in that, S4 specifically includes: S401: Input the metadata semantic graph and hybrid index set of the multimodal metadata semantic space into the representation learning and decision optimization module; S402: Iteratively aggregate the nodes of the metadata semantic graph using a graph neural network to obtain the node embedding set and the matching score of the task participants; S403: Determine the constraint profile of the participants based on standard metadata; S404: Based on the matching scores of the task participants and the constraint profiles of the participants, output the optimal collaborative computing alliance and execution strategy through deep reinforcement learning; S405: Based on the optimal collaborative computing alliance and the execution strategy, select the participants suitable for the task.

5. The multimodal data secure sharing method according to claim 1, characterized in that, S5 specifically includes: S501: Construct a dynamic container model, wherein the dynamic container model specifically includes: templates, instances, and a dynamic container tree; S502: Map the multimodal data to the fields of the template to form the instance; S503: Assign registration numbers to the template and the instance respectively to obtain template identifier and instance identifier; S504: Based on the template identifier, the instance identifier, and the dynamic container tree, generate a path directory, a shard set, and a multimodal auxiliary index using a path navigation function; S505: Parses natural language queries into multimodal analysis query language statements through a large language model; S506: The multimodal analysis query language statement is decomposed into semantically independent subtasks through the path navigation function; S507: Construct a dependency graph using the semantically independent subtasks as nodes and data dependencies as edges; and extract the cost features of the semantically independent subtasks. S508: Based on the semantically independent subtasks, the dependency graph, and the cost features, a job description is generated through a self-optimizing data processing plan. S509: According to the job description, execute the AI ​​ready output and loading specifications to obtain the search results.

6. The multimodal data secure sharing method according to claim 5, characterized in that, Specifically, S508 includes: S5081: Based on the semantically independent subtasks, the dependency graph, and the cost features, a parallel execution plan and scheduling strategy are obtained through a cost model optimizer; S5082: By cleaning the primitive library and quality metric function across modalities, the cleaning process is formalized as a directed acyclic graph; S5083: Based on the directed acyclic graph, reinforcement learning is used to model the cleaning arrangement as a Markov decision process to obtain the optimal directed acyclic graph for cleaning, the parameter set, and the cleaning plan. S5084: Assemble the optimal cleaning directed acyclic graph, the parameter set, the cleaning plan, the parallel execution plan, and the scheduling strategy into the job description.

7. The multimodal data secure sharing method according to claim 1, characterized in that, S6 specifically includes: S601: Construct a candidate neural network model library based on the directed acyclic graph; S602: Construct a search space in the candidate neural network model library using reusable functional blocks as the smallest unit; S603: Based on the search space, select a set of candidate subnets using a controller that employs reinforcement learning or evolutionary search; S604: Construct a policy controller; S605: Evaluate the candidate subnet set using the multi-objective reward function and constraint set in the reinforcement learning process; S606: Based on the evaluation results, update the strategy controller to obtain the polymer network set and the evaluation set; S607: Based on the polymer network set, assemble candidate fusion models according to the fusion topology; S608: Introduce a shared parameter supernetwork and a weight shared index to perform hot start and light fine-tuning on the candidate fusion model, obtain the evaluation results, and register the evaluation results to the subnet storage pool and performance cache; S609: Based on the candidate fusion model, the weight sharing index, the subnet storage pool, and the performance cache, the neural network model is obtained by aggregating gradients across multiple subnets.

8. The multimodal data secure sharing method according to claim 7, characterized in that, Specifically, S609 includes: S6091: Based on the evaluation results, update the weights of the shared operations by aggregating gradients through the multi-subnet: ； ； in, This represents the gradient after aggregation. ω s This represents the gradient weight factor of the subnet. This represents the loss gradient of the subnet. W op The weight parameter represents the weight of the sharing operation. η Indicates the learning rate; S6092: Combining the weights of the updated shared operations, and based on the performance cache, identify frequently used modules and high-contribution modules; S6093: Based on the statistical results, construct a set of priority paths and update the set of priority paths; S6094: Based on the updated priority path set, combined with knowledge distillation or regression distillation, the neural network model is obtained.

9. The multimodal data secure sharing method according to claim 1, characterized in that, Specifically, S7 includes: S701: Update the search results based on the domain knowledge base and the knowledge graph; S702: Update the hyperparameters of the neural network model based on the domain prior distribution and constraints in the domain knowledge base.

10. A multimodal data secure sharing system, characterized in that, include: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the multimodal data secure sharing method as described in any one of claims 1 to 9.

Images