Decision-making method and device based on multi-modal data, equipment and medium

Abstract

The invention relates to the technical field of intelligent decision making, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a decision making method, device, equipment and medium based on multi-modal data, comprising: collecting and cleaning multi-modal original data, executing format conversion and desensitization processing, and generating standardized data; multi-modal features are extracted and fused based on the standardized data, and multi-modal feature vectors are generated; performing reasoning on the multi-modal feature vector through a reasoning model to generate reasoning result data; pushing the reasoning result data to a review terminal, and receiving correction feedback data; and updating parameters of the reasoning model based on the corrected feedback data, and generating an updated reasoning model. According to the method, after data cleaning, format conversion and desensitization processing, multi-modal feature extraction and reasoning model reasoning processing are executed, effective fusion and accurate reasoning of data multi-modal features are achieved, and decision accuracy and data safety are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making technology, and in particular to a decision-making method, device, equipment and storage medium based on multimodal data. Background Art

[0002] In the healthcare sector, artificial intelligence technology has gradually been applied to auxiliary diagnosis and personalized treatment. However, existing technologies still have many shortcomings in data collection and processing. First, medical data is collected from diverse sources and in different formats. Real-time time series data, medical imaging data, and text data are usually scattered across different systems. There is a lack of unified collection and processing standards, which makes it difficult to effectively integrate data. Second, the data cleaning and labeling process is complex. The raw data often contains noise, missing values, and redundant information, which affects data quality and further reduces the training effect of the model. In addition, medical data contains a large amount of sensitive information, such as patient identity, medical history records, and diagnostic reports. Existing technologies are insufficient in data desensitization and privacy protection, which can easily lead to patient privacy leaks.

[0003] In the fintech sector, artificial intelligence (AI) technology is widely used in credit risk assessment and fraud detection. However, existing technologies still face numerous challenges in data processing. First, financial data comes from diverse sources, including transaction data, user behavior data, and unstructured text data. These data formats and structures vary, lacking unified standards, making data preprocessing and format conversion difficult. Second, data often contains sensitive information, such as user identity, transaction records, and credit scores. Without effective desensitization and encryption measures, data leaks can easily occur, jeopardizing user privacy and system security. Furthermore, existing technologies also have shortcomings in data cleaning and labeling. Improper handling of outliers and missing values in the data can lead to poor model training results.

[0004] In the field of multimodal data processing, existing technologies mainly rely on data format conversion, feature extraction and feature fusion methods. However, existing technologies have the following deficiencies in the process of multimodal data processing. First, the data formats of different modalities differ significantly. There are large differences in data representation and feature extraction between time series data, image data and text data. Traditional data preprocessing methods are difficult to efficiently convert these data into a unified format. Secondly, there is a lack of standardized methods for desensitizing sensitive information in multimodal data. For example, privacy attributes in time series data, identity information in image data and sensitive fields in text data are often processed by simple masking or hashing in existing technologies, which cannot achieve efficient and reliable privacy protection. In addition, existing technologies also lack automation and intelligent means in the data cleaning and labeling links. Noise and incorrect labeling in the data will directly affect the training effect of downstream models, and thus affect the accuracy of reasoning. Summary of the Invention

[0005] The main purpose of the present invention is to provide a decision-making method, device, equipment and storage medium based on multimodal data, aiming to solve the technical problems in the existing technology of lacking unified standards for data collection, format conversion and desensitization processing, resulting in low data quality and insufficient privacy protection, which affects the effective integration of multimodal data and the accuracy of intelligent decision-making.

[0006] To achieve the above objectives, the present invention provides a decision-making method based on multimodal data, comprising:

[0007] Collecting original multimodal data, and performing cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data;

[0008] Performing format conversion and desensitization processing on the cleaned and annotated data to generate standardized data;

[0009] Performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector;

[0010] Performing reasoning processing based on the multimodal feature vector through an inference model to generate inference result data;

[0011] Pushing the inference result data to a review terminal and receiving correction feedback data returned by the review terminal;

[0012] The parameters of the inference model are updated based on the corrected feedback data to generate an updated inference model.

[0013] Furthermore, to achieve the above-mentioned object, the present invention provides a decision-making device based on multimodal data, comprising:

[0014] A data preprocessing module is used to collect original multimodal data and perform cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data;

[0015] A data standardization module is used to perform format conversion and desensitization processing on the cleaned and annotated data to generate standardized data;

[0016] A multimodal feature extraction module, configured to perform multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector;

[0017] An inference model module, configured to perform inference processing based on the multimodal feature vector through an inference model to generate inference result data;

[0018] A feedback collection module, configured to push the inference result data to a review terminal and receive correction feedback data returned by the review terminal;

[0019] The model updating module is used to update the parameters of the inference model based on the corrected feedback data to generate an updated inference model.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a decision-making program based on multimodal data stored in the memory and executable on the processor. When the decision-making program based on multimodal data is executed by the processor, the steps of the decision-making method based on multimodal data as described above are implemented.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a decision-making program based on multimodal data is stored. When the decision-making program based on multimodal data is executed by a processor, the steps of the decision-making method based on multimodal data as described above are implemented.

[0022] Beneficial effects: The present invention relates to the field of intelligent decision-making technology and can be applied to business scenarios such as financial technology and medical health. It discloses a decision-making method, device, equipment and medium based on multimodal data, including: collecting original data and performing cleaning and labeling processing to generate cleaned and labeled data; performing format conversion and desensitization processing on the cleaned and labeled data to generate standardized data; performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector; performing inference processing based on the multimodal feature vector through the inference model to generate inference result data; pushing the inference result data to the review terminal and receiving the corrected feedback data returned by the review terminal; updating the parameters of the inference model based on the corrected feedback data to generate an updated inference model. The present invention realizes the effective fusion and precise reasoning of the multimodal features of the data by performing multimodal feature extraction and inference model inference processing after data cleaning, format conversion and desensitization processing, significantly improving the accuracy of decision-making and data security, and ensuring the privacy protection of data and dynamic model updates throughout the entire process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 Schematic diagram of an application environment of a decision-making method based on multimodal data in one embodiment of the present invention;

[0025] Figure 2 1 is a flow chart of an embodiment of a decision-making method based on multimodal data according to the present invention;

[0026] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a decision-making device based on multimodal data of the present invention;

[0027] Figure 4A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0028] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] The decision-making method based on multimodal data provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can collect raw data from the user terminal and perform cleaning and annotation processing to generate cleaned and annotated data; perform format conversion and desensitization processing on the cleaned and annotated data to generate standardized data; perform multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector; perform inference processing based on the multimodal feature vector through the inference model to generate inference result data; push the inference result data to the review terminal and receive the corrected feedback data returned by the review terminal; update the parameters of the inference model based on the corrected feedback data to generate an updated inference model. The present invention realizes the effective fusion and accurate inference of multimodal features of data by performing multimodal feature extraction and inference model inference processing after data cleaning, format conversion and desensitization processing, significantly improves the accuracy of decision-making and data security, and ensures data privacy protection and dynamic model update in the whole process. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of a decision-making method based on multimodal data provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the decision-making method based on multimodal data proposed in the present invention includes the following steps:

[0033] S10, collecting original multimodal data, and performing cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data;

[0034] In this embodiment, raw data is collected and cleaned and annotated to ensure the quality and accuracy of subsequent multimodal data. Raw data can be collected in various forms, such as time series data acquired by sensor devices, raw image data acquired by image acquisition devices, and unstructured text data received through text interfaces.

[0035] Sensor devices can be physiological monitoring devices, environmental monitoring devices, or smart wearable devices. Time series data typically includes timestamps and corresponding physical quantity data, such as heart rate, blood pressure, and temperature. The data collection process can present issues such as inconsistent sampling frequencies, sampling delays, or data loss. Therefore, sampling rate alignment is necessary. Sampling rate alignment can be achieved through interpolation algorithms and resampling methods to ensure consistency across all timestamp data.

[0036] Raw image data typically comes from cameras, medical imaging devices (such as CT and MRI), or remote sensing equipment. This image data may have differences in resolution, color space, and image noise due to different acquisition devices or environments. To ensure the consistency and usability of image data, resolution normalization is required. This can include bilinear interpolation or bicubic interpolation to ensure that all image data has the same resolution. Furthermore, for image data with high noise, median filtering or convolution noise reduction can be used.

[0037] Unstructured text data can be accessed through APIs from channels such as medical records, customer service conversations, and news articles. Due to its complex formats and diverse encodings, unstructured text data requires encoding conversion to ensure that all text data uses a unified character encoding format (such as UTF-8). Named entity recognition is a key step in text data. Pre-trained language models (such as BERT or GPT models) can be used to extract key named entities from text, such as names of people, places, and medical terms, and further identify the relationships between entities.

[0038] To ensure data structuring and standardization, the extracted entity objects are mapped to standardized nodes in the domain knowledge graph. Domain knowledge graphs can include medical knowledge graphs (diseases, symptoms, and drug relationships) and financial knowledge graphs (customer attributes, financial products, and transaction relationships). Through knowledge graph mapping, standardized semantic labels can be assigned to entity objects, ensuring consistency and interpretability in subsequent data processing.

[0039] In time series data, due to the possibility of data loss during the acquisition process (e.g., device signal loss, network interruption), interpolation algorithms are needed to fill in the missing segments. Interpolation algorithms can calculate missing values based on the adjacent data points, such as linear interpolation, spline interpolation, or K-nearest neighbor interpolation.

[0040] For missing areas in image data (such as those in medical images), generative inpainting models can be used to complete them. Generative inpainting models can automatically predict and fill in missing areas using methods based on convolutional neural networks (CNNs) or generative adversarial networks (GANs).

[0041] Ultimately, the complete time series data, restored image data, and structured annotation data need to be integrated into a multimodal cleaned and annotated dataset. This integration ensures that each data type remains independent while ensuring cross-modal data consistency through data indexes or data IDs.

[0042] In one specific embodiment, during the data collection phase, data collection can be achieved by deploying multiple types of sensors and collection devices. For time series data, smart wearable devices, industrial sensors or server monitoring devices can be used for collection, and the data consistency can be ensured by an automatic sampling rate adjustment mechanism. For image data, it can be collected by high-definition cameras or medical imaging equipment (such as CT, MRI), and resolution standardization and noise filtering can be performed at the collection end. For text data, third-party text data sources (such as electronic medical records, customer service records) can be called through API interfaces and extracted in real time. During the data cleaning and annotation phase, edge computing devices or cloud computing platforms can be used for data preprocessing. For the interpolation of time series data, linear interpolation (fast but low accuracy) or spline interpolation (computationally complex but high accuracy) can be selected according to the specific application. In image data repair, a generative repair model (such as an image repair model based on U-Net or GAN) can be used, and combined with domain knowledge (such as organ structure information in medical images) to improve the repair effect. In named entity recognition and knowledge graph mapping of text data, pre-trained language models (such as BERT or GPT) and domain knowledge graphs (such as medical knowledge graphs or financial knowledge graphs) can be combined to achieve semantic understanding and standardization. During the data integration phase, multimodal data can be indexed by unique IDs or timestamps to ensure consistency across modal data.

[0043] Example: In the healthcare sector, smart wearable devices collect time-series data such as a patient's heart rate, blood pressure, and blood oxygen saturation. High-definition cameras capture facial images of patients, and electronic medical record systems extract patient medical histories and examination report text. This method cleans and annotates the collected multimodal data. Linear interpolation is used to fill missing values in heart rate data, and a GAN inpainting model is used to repair damaged areas in facial images. Named entity recognition is used to extract disease names and medication information from medical record text. These information is then mapped to standardized nodes in a medical knowledge graph, generating structured data with semantic labels.

[0044] In the financial sector, smart devices collect user transaction behavior data (such as transaction time, transaction amount, and payment method), user profile images, and customer service conversation text. Time series data is aligned through sampling rate alignment to ensure time series consistency, image data is normalized for resolution consistency, and text data is extracted through named entity recognition to extract customer identity information and transaction types. This customer information is then matched with standard customer types in the financial knowledge graph to generate structured data with semantic labels.

[0045] This implementation achieves standardization and consistency of multimodal data through data collection, cleaning and annotation, and structured processing. Differences between data sources are eliminated through sampling rate alignment, resolution standardization, and text encoding conversion. Named entity recognition and knowledge graph mapping ensure the semantic accuracy of text data. Interpolation and generative repair fill in missing data areas.

[0046] S20, performing format conversion and desensitization processing on the cleaned and annotated data to generate standardized data;

[0047] In this embodiment, after the cleaned and annotated data is generated, format conversion and desensitization are important steps to ensure data standardization and privacy protection. The cleaned and annotated data may include time series data, image data, and text data. Each data type has different formats and privacy protection requirements, so format conversion and desensitization must be performed separately for each data type.

[0048] For time series data, format conversion first requires ensuring the consistency of the timestamp format of the time series data. This can be achieved by adopting the ISO 8601 format (e.g., "YYYY-MM-DDThh:mm:ssZ"). Standardized timestamps ensure that the data can be unambiguously identified and used across multiple systems. Furthermore, time series data may have inconsistent sampling frequencies, necessitating time series resampling. Resampling can utilize linear interpolation, spline interpolation, or nearest neighbor interpolation to generate time series data with a uniform sampling frequency.

[0049] Desensitization ensures that sensitive information contained in time series data (such as patient physiological data and transaction amounts) is not leaked. For sensitive fields in time series data, differential privacy noise can be applied. This introduces random noise to the original values, ensuring that the original information cannot be deduced from the values. Noise can be generated using a Gaussian or Laplace distribution, with specific parameters set based on the application scenario and privacy requirements.

[0050] For image data, format conversion first involves uniform color space conversion, such as converting RGB images to YUV or grayscale formats. This color space conversion facilitates subsequent image processing and feature extraction. Next, image tensor normalization involves normalizing image pixel values and mapping them to a predefined numerical range (e.g., [0, 1]). This not only facilitates subsequent deep learning model training but also prevents numerical overflow.

[0051] Image data desensitization primarily targets private areas, such as faces and recognizable features (such as license plates and lesion areas). This can be accomplished using pixel blurring, mosaic occlusion, or Gaussian blurring. Pixel blurring reduces image resolution to hide details; mosaic occlusion covers private areas with low-resolution blocks; and Gaussian blurring smoothes private areas using a convolution kernel, ensuring irreversibility.

[0052] For text data, format conversion first ensures that the text data uses a unified character encoding (such as UTF-8). The text data may contain structured annotations, such as named entities (personal names, addresses, organization names) and labels (such as disease names and transaction types). These text annotations must be mapped to a predefined classification code table to ensure semantic consistency. Predefined classification code tables can include medical coding tables (such as ICD-10) or financial product classification tables (such as the financial product type table).

[0053] Desensitization of text data primarily targets annotated data that poses privacy risks. Privacy attribute fields, such as names, ID numbers, and phone numbers, can be replaced through hash encryption. Hash encryption uses an irreversible hash algorithm (such as SHA-256), making it impossible to recover the original information even if a malicious user obtains the encrypted text. Furthermore, tokenization can be used to further enhance privacy protection by replacing private information with randomly generated, meaningless identifiers.

[0054] After format conversion and desensitization, various types of data are integrated into standardized data to ensure that they have consistent formats, the same level of privacy protection, and high-quality usability in subsequent processing. Standardized data includes the following three types:

[0055] Time series data in a unified format: standardized timestamps, consistent sampling frequency, and differential privacy masking;

[0056] Standardized image tensor: color space unification, pixel normalization, and privacy area blurring;

[0057] Uniformly encoded text data: consistent character encoding, standardized semantic tags, and hash encryption of privacy fields.

[0058] In practical applications, different format conversion and desensitization techniques can be selected based on different data sources and application scenarios. In the IoT environment, time-series data collected by sensors can be sampled and desensitized in real time using edge computing devices, and differential privacy protection can be applied before transmission to the cloud. Noise distribution parameters can be dynamically adjusted based on the privacy budget. In medical image processing systems, medical image data (such as CT and MRI) undergoes color space conversion and normalization immediately after acquisition. Faces or lesion areas in the image are detected using an automatic recognition algorithm, and Gaussian blurring is used for privacy protection. Image files are encrypted before being saved to the database. In text data processing systems, natural language processing (NLP) models (such as BERT or GPT) can be combined to automatically identify named entities (such as names, addresses, and disease names) in text and encrypt this sensitive information using a hashing algorithm. For medical records, the ICD-10 coding table can be used to standardize diagnostic labels; for customer data in the financial sector, a predefined financial product classification table can be used for labeling. During the data integration process, various data types are indexed by unique identifiers (such as data IDs or timestamps) to ensure their consistency and traceability in subsequent multimodal processing stages.

[0059] Example: In the healthcare field, patients' physiological monitoring data (such as heart rate and blood pressure) are collected through smart wearable devices, and sampling rate alignment and differential privacy processing are performed in real time to ensure data continuity and privacy protection. Patients' facial images are collected by high-resolution cameras and pixel-blurred before storage to ensure that facial features are unrecognizable. Electronic medical record text data uses named entity recognition to extract patient information (such as name, medical history, and diagnosis results), and hash encryption is used to replace sensitive information. The final standardized data generated includes desensitized time series data, anonymized image data, and encrypted text data.

[0060] In the financial sector, customer transaction data (such as transaction time and amount) is collected in real time through financial systems. Format conversion ensures that all timestamps are in UTC format, and transaction amounts are protected through differential privacy noise. Customer profile pictures are captured and mosaiced before upload. The text of customer conversations with customer service representatives is processed using a natural language processing model to extract customer identity information and transaction type, and hash encryption is used to replace customer information. The resulting standardized data includes desensitized transaction time series data, anonymized profile picture images, and encrypted text data.

[0061] This embodiment can achieve the standardization of multimodal data while ensuring data security and privacy through format conversion and desensitization processing. Format conversion ensures that different data types (time series, images, text) maintain consistency in numerical range, format structure, and semantic labels, eliminating format differences between multi-source data. Desensitization processing ensures that sensitive information in the data is effectively protected, reducing the risk of data leakage. The resulting standardized data is of high quality and high security, providing a reliable foundation for subsequent feature extraction and model training.

[0062] S30, performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector;

[0063] In this embodiment, multimodal feature extraction and fusion processing converts multiple data types (such as time series data, image data, and text data) into a unified feature representation, thereby enabling collaborative analysis of multi-source information. The generation of multimodal feature vectors relies on specific deep learning models and feature extraction methods to ensure that information from each data type is effectively captured and represented.

[0064] First, for the image data in the standardized data, feature extraction is performed using a three-dimensional convolutional neural network (3D-CNN). 3D-CNN is a neural network architecture that can capture spatial and depth features and is suitable for medical images (such as CT, MRI) or video frame sequences. The image data is first converted into a standardized image tensor and input into the convolution layer of the 3D-CNN. The convolution layer uses multiple three-dimensional convolution kernels to perform convolution operations on the input image to extract low-level features (such as edges and textures). The pooling layer reduces the size of the feature map through downsampling operations while retaining important information. After multiple layers of convolution and pooling, an image feature vector is finally generated, which retains the spatial information and contextual semantics of the image data.

[0065] Secondly, for the time series data in the standardized data, feature extraction is performed using a Long Short-Term Memory (LSTM) network. LSTM is a classic recurrent neural network (RNN) structure designed specifically for processing time series data and has the ability to capture long-term dependencies. The time series data is first formatted into a unified time series tensor and input into the LSTM layer in time steps. The memory units in the LSTM can capture both short-term and long-term dependencies in the data, such as changes in a patient's physiological indicators or the frequency of financial transactions. The final output time series feature vector can represent the dynamic patterns and temporal correlations in the time series data.

[0066] For the text data in the standardized dataset, pre-trained language models (such as BERT or GPT) are used for contextual encoding. These language models use a self-attention mechanism to understand the contextual semantics of the text. The text data is first tokenized and encoded into word vectors, which are then input into the encoder layer of the language model. The self-attention mechanism assigns weights based on the relevance between each word, ensuring that the text features capture semantic dependencies and sentence structure. The resulting text feature vector not only represents the text content but also preserves the semantic information within the text.

[0067] After extracting unimodal features, the feature vectors of each modality (image feature vectors, time series feature vectors, and text feature vectors) must be mapped to the same latent space dimension to ensure they have the same representation space during the subsequent fusion process. Latent space mapping can be achieved through a fully connected layer or linear transformation, where each feature vector is projected to the same feature dimension.

[0068] Multimodal feature fusion employs a cross-modal attention mechanism, which is key to ensuring effective interaction between features from each modality. The cross-modal attention mechanism calculates the correlation between feature vectors from each modality and assigns different interaction weights. Specifically, image feature vectors, time series feature vectors, and text feature vectors are input into the attention layer as queries, keys, and values. The correlation weights between them are calculated, and a weighted sum operation is performed to generate a weighted fused feature vector. This process ensures the effective transfer and integration of information between modalities.

[0069] Finally, the weighted fused feature vector is input into the dimensionality reduction and normalization processing module. Dimensionality reduction can be achieved through principal component analysis (PCA) or adaptive pooling, aiming to reduce redundant information and improve feature compactness. Normalization ensures that the numerical range of the feature vector is consistent, for example, by normalizing the vector length to unit length through L2 regularization. The resulting multimodal feature vector is an efficient feature representation that can comprehensively represent information from image, time series, and text data.

[0070] In practical applications, multimodal feature extraction and fusion processing can adopt different feature extraction models and fusion strategies according to different application scenarios:

[0071] In medical imaging diagnosis, 3D-CNN can use pre-trained ResNet3D or DenseNet3D models to extract spatial features from CT or MRI images. Time series data (such as electrocardiograms and blood pressure waveforms) can be captured using a bidirectional LSTM model. Text data (such as electronic medical records) can be semantically encoded using the BERT or BioBERT models, with multi-head attention mechanisms employed for multimodal fusion.

[0072] In financial risk assessment, image data can include customer ID photos or handwritten signatures, using 2D-CNN for feature extraction. Time series data can include customer transaction records or credit score time series, modeled using a one-dimensional convolutional neural network (1D-CNN) or LSTM. Text data can include customer contracts or transaction descriptions, semantically encoded using GPT or pre-trained language models in the financial field. During the multimodal fusion stage, weighted averaging or dynamic attention mechanisms can be used to assign feature weights.

[0073] In multimodal emotion recognition, image data can be facial expression images, using facial expression recognition models (such as VGG-Face) to extract expression features. Time series data can be speech waveforms, using LSTM or GRU to extract speech features. Text data can be the result of user speech-to-text conversion, using BERT to extract text semantics. Multimodal feature fusion can be achieved by combining self-attention and channel attention.

[0074] Example: In the healthcare sector, patients' medical images (such as CT scans), electrocardiograms (ECGs) (time series data), and electronic medical records (text data) are collected simultaneously. CT images are used to extract spatial features using a 3D-CNN, ECGs are used to extract temporal-dependent features using an LSTM, and EMRs are used to extract semantic features using a BERT. These three feature vectors are fused using a cross-modal attention mechanism to generate a weighted fused feature vector. This vector is then processed through dimensionality reduction and normalization to generate a multimodal feature vector. This feature vector is then used for subsequent disease classification and risk prediction, simultaneously referencing image, time series, and text information.

[0075] In the financial sector, customer transaction records (time series data), ID photos (image data), and transaction contracts (text data) are collectively collected. Transaction records are processed using LSTM to extract transaction behavior features, ID photos are processed using 2D-CNN to extract identity features, and contract terms and conditions are extracted using BERT. These three feature vectors are then mapped to the same latent space and fused using a dynamic attention mechanism to generate a multimodal feature vector. This feature vector is used for customer credit scoring and risk assessment, integrating customer behavior, identity, and contract information.

[0076] This embodiment can effectively integrate information from multiple data sources through multimodal feature extraction and fusion processing to generate a unified multimodal feature vector. This feature vector can not only retain the key information in each data type (such as the spatial characteristics of images, the dynamic patterns of time series data, and the semantic relationships of texts), but also capture the correlation between different modalities through a cross-modal attention mechanism, thereby improving the decision-making accuracy and robustness of the model. Especially in tasks involving complex information (such as medical diagnosis and financial risk control), multimodal feature fusion can significantly improve the recognition and prediction performance of the model.

[0077] S40, performing reasoning processing based on the multimodal feature vector through an inference model to generate inference result data;

[0078] In this embodiment, inference processing is a decision-making process based on multimodal feature vectors. Its core lies in using a pre-trained multi-task joint learning model to perform inference calculations to generate inference result data. The multi-task joint learning model is a deep learning model that integrates multiple tasks and can simultaneously handle classification, regression, and causal reasoning tasks. This model can generate decision results from multiple task perspectives based on the input of multimodal feature vectors, ensuring the comprehensiveness and accuracy of the inference results.

[0079] First, the reasoning process begins by loading a pre-trained multi-task joint learning model. The model contains a classification task module, a regression task module, and a causal reasoning module. The classification task module is based on a fully connected layer (Fully Connected Layer) or a convolutional neural network (CNN), which can output category labels based on the input feature vector, such as disease diagnosis type, risk level, or customer credit rating. The regression task module usually uses a linear layer or a multi-layer perceptron (MLP) to generate continuous numerical prediction results, such as disease risk probability or credit score. The causal reasoning module is based on a causal graph model or a self-attention mechanism to infer the causal relationship between variables from multimodal features.

[0080] Next, the multimodal feature vector is input into the multi-task joint learning model. The classification task module first processes the feature vector, performs feature mapping on the input features through a multi-layer neural network, and generates category labels. Category labels are usually discrete variables, such as "pneumonia", "diabetes", or "high-risk customers", and the module is able to output the probability distribution of each category. The regression task module performs continuous numerical predictions based on the same feature vector and outputs one or more predicted values, such as the probability of a patient being ill or the customer's credit score. The causal reasoning module captures the causal relationship in the feature vector through a graph structure or a self-attention mechanism to generate a causal path. This path describes the causal dependency between the features, such as "smoking leads to decreased lung function" or "high income reduces the risk of default."

[0081] The third core step in reasoning processing is causal correction. Causal correction involves logically modifying classification and regression results based on the causal path output by the causal inference module. This correction ensures that the inference results conform not only to statistical laws but also to domain knowledge or logical rules. For example, if the causal path indicates that "high blood pressure causes heart disease," but the classification task module predicts "no heart disease," the causal correction module can adjust the classification result to "possible heart disease," thereby ensuring the legitimacy of the inference result.

[0082] After causal correction is completed, the reasoning process also includes knowledge graph verification. The knowledge graph is a structured domain knowledge base that contains entities and the relationships between them (such as the triple of disease-symptom-treatment). The category labels, continuous prediction values, and causal relationship paths in the reasoning results will be matched and verified with the association rules in the knowledge graph. If the reasoning result conforms to the rules in the knowledge graph, it is considered to have passed the verification; if it does not conform, it may be marked as an anomaly or corrected. Knowledge graph verification can not only verify the rationality of the reasoning results, but also further improve the reliability of the reasoning results.

[0083] Finally, the inference processing generates structured output data. Structured inference output data typically includes category labels (such as predicted disease type or customer credit rating), continuous predicted values (such as disease risk probability or credit score), and causal paths (such as causal dependencies between features). These inference results can be further formatted into JSON, XML, or other structured formats for subsequent display, analysis, or storage.

[0084] Example: In the healthcare sector, patients' medical images (such as CT scans), physiological indicators (such as blood pressure and blood sugar), and electronic medical records (such as symptom descriptions) are collected simultaneously to generate a multimodal feature vector. The inference model processes this feature vector through a multi-task joint learning model, outputting a disease classification label (such as "pneumonia"), a risk prediction value (such as "90% probability of disease"), and a causal relationship (such as "smoking → decreased lung function → pneumonia"). The inference results are verified against the medical knowledge graph (such as disease-symptom-treatment rules) to ensure the rationality of the results.

[0085] In the financial sector, customer transaction records (time series data), ID photos (image data), and loan contracts (text data) are integrated into multimodal feature vectors. The inference model outputs a credit rating (e.g., "medium risk"), a default probability (e.g., "15%"), and a causal path (e.g., "low income → high debt → high default risk"). These inference results are verified using the financial knowledge graph (e.g., credit rating rules and the relationship between income and default) to ensure compliance with financial regulations.

[0086] This embodiment performs reasoning processing based on multimodal feature vectors through an inference model, and can realize joint reasoning of three tasks: classification, regression, and causal reasoning within the same model framework. This multi-task joint learning can not only improve the efficiency of reasoning, but also ensure the multi-dimensional consistency of the reasoning results. The combination of causal correction and knowledge graph verification ensures that the reasoning results not only conform to statistical laws, but also conform to domain knowledge, thereby improving the reliability and interpretability of the reasoning results. Through the structured output of the reasoning result data, a variety of reasoning information can be standardized, which is convenient for subsequent analysis and application.

[0087] S50, pushing the inference result data to a review terminal, and receiving correction feedback data returned by the review terminal;

[0088] In this embodiment, the process of pushing inference result data to the review terminal and receiving correction feedback data aims to improve the accuracy and interpretability of inference results through automated or manual review mechanisms. This process technically covers four core links: data transmission, security assurance, and review and feedback collection. It also supports flexible switching between automated and manual review.

[0089] First, after the inference result data is generated, it is pushed to the review terminal. The core of this push lies in data transmission and security. Data transmission typically uses encrypted transmission protocols (such as TLS / SSL) to ensure the confidentiality and integrity of data during network transmission. The inference result data can be in structured data formats such as JSON, XML, or Protobuf to ensure that the data can be quickly parsed and displayed by the terminal device. The target terminal for push can be a doctor's workstation, a financial analyst's console, an automated review system, or a decision support terminal in other fields.

[0090] In the review terminal, the inference result data will be presented in a visual interface or automated rule engine. For manual review mode, the interface design should include classification labels (such as "pneumonia" and "high-risk customers"), continuous prediction values (such as "probability of illness 85%" or "credit score 720"), and causal paths (such as "smoking → decreased lung function → pneumonia" or "low income → high default risk"). The terminal should provide user interaction functions to allow reviewers to correct or supplement the inference results, such as modifying category labels, adjusting predicted probabilities, or correcting causal relationships.

[0091] In automated review mode, the inference result data is pushed to a rule engine or automated audit module. This module can automatically detect anomalies or unexpected parts in the inference result data based on predefined rules (such as threshold rules and conditional judgment rules). For example, if the "probability of illness" in a medical diagnosis exceeds 90% and the symptoms are severe, it will be automatically marked as "high risk" and the diagnosis label will be revised. In financial risk assessment, if the "credit score" is less than 500 and the income is below the preset threshold, it will be automatically marked as a "high-risk customer."

[0092] Correction feedback data can be generated manually or automatically. Manual correction feedback data typically includes the corrected classification label, the corrected continuous prediction value, and the corrected causal path. Automated correction feedback data, on the other hand, is automatically generated based on a rules engine and includes the corrected classification label and the corrected prediction value.

[0093] Correction feedback data must also be secure during transmission. End-to-end encryption (such as TLS / SSL) or digital signatures (such as RSA) are typically used to ensure that feedback data has not been tampered with. To ensure the reliability of correction feedback data, a unique feedback ID can be generated for each feedback, recording the feedback time, review method (manual or automated), review rules (such as rule ID), and correction content. This correction feedback data will serve as an important reference for subsequent model updates and optimizations.

[0094] For example, in a medical diagnosis scenario, the inference result data is pushed to the doctor's workstation. In manual review mode, the doctor can view the patient's imaging diagnosis results, disease classification labels, and risk prediction probabilities. The doctor can manually correct the classification label (such as "pneumonia" to "bronchitis") or adjust the prediction probability (such as "85%" to "60%"). The corrected feedback data is uploaded to the server via encrypted transmission. In the medical automated review scenario, the inference result data is pushed to the automated rule engine. The rule engine automatically checks whether the "probability of illness" exceeds 90%. If it exceeds and the symptoms are severe, it automatically corrects the diagnosis label to "high risk" and records the reason for the correction. The corrected feedback data is automatically uploaded to the server.

[0095] In financial credit assessment, the inference result data is pushed to the financial analyst's terminal. In manual review mode, the analyst can view the customer's credit rating, default probability, and causal path. The analyst can manually adjust the credit rating (e.g., "medium risk" to "high risk") or modify the causal path (e.g., "low income → high debt → high default risk"). The correction feedback data is uploaded to the server via encrypted transmission. In the financial automated review scenario, the inference result data is pushed to the automated audit module. The rule engine automatically checks whether the "credit score" is less than 500 and the income is below the preset threshold. If the conditions are met, it is automatically marked as a "high-risk customer" and the credit rating is modified. The correction feedback data is automatically uploaded to the server.

[0096] In industrial fault diagnosis, the inference result data is pushed to the engineer's terminal. In manual review mode, engineers can view the equipment fault classification (such as "mechanical fault"), the cause of the fault (such as "wear and tear"), and the predicted repair time (such as "within 48 hours"). Engineers can correct the fault type (such as "electrical fault") or adjust the repair time (such as "within 24 hours"), and upload the corrected feedback data to the server. In industrial automation diagnosis, the inference result data is pushed to the automated fault rule engine. The rule engine automatically checks the fault category and risk level. If the repair time exceeds 48 hours and the fault has a significant impact, the fault type is automatically corrected to "high-risk fault." The corrected feedback data is automatically uploaded to the server.

[0097] This embodiment can realize a human-machine collaborative and automated intelligent decision-making mechanism by pushing the inference result data to the review terminal and receiving correction feedback data. The results generated by the inference model can be verified and corrected through manual review or automated rule engine to ensure the accuracy and reliability of the inference results. In the manual review mode, professionals can correct the inference results based on actual conditions to improve the professionalism and credibility of the decision. In the automated review mode, the rule engine can automatically detect and correct the inference results based on predefined rules, improve the review efficiency and reduce the manual workload. The correction feedback data can be further used for dynamic optimization and continuous training of the model to achieve adaptive updating of the model.

[0098] S60: Update the parameters of the inference model based on the corrected feedback data to generate an updated inference model.

[0099] In this embodiment, the process of updating the inference model's parameters based on the corrected feedback data and generating the updated inference model is key to achieving model adaptability and dynamic optimization. This process encompasses six technical features: obtaining corrected feedback data, data preprocessing, selecting a model update strategy, updating model parameters, performance verification, and registering the new model version. Each feature directly impacts the model's optimization effectiveness and stability.

[0100] First, correction feedback data is the correction information generated by the review terminal. It can come from manual review or automated review modules. Correction feedback data typically includes the corrected classification labels, corrected continuous prediction values, and corrected causal paths. This feedback data is of great guiding significance and can provide direct basis for model optimization. After receiving the correction feedback data, it is stored in the data processing module and becomes the basic data source for subsequent incremental training.

[0101] During the preprocessing phase of the correction feedback data, data deduplication and label alignment are first performed to ensure data quality and consistency. Data deduplication removes duplicate correction feedback data to prevent data redundancy that can lead to unstable model updates. Label alignment ensures that the label format in the correction feedback data is consistent with the label format used for model training. For example, in the medical field, "pneumonia" is aligned to the standard code "J18.0," and in the financial field, "high risk" is aligned to "high default risk."

[0102] During the model update strategy selection phase, the Elastic Weight Consolidation (EWC) algorithm is employed to ensure that the model retains existing knowledge during incremental training. Elastic Weight Consolidation is an incremental learning strategy that assigns dynamic weights to model parameters, protecting important parameters from over-adjustment during the update process. This approach effectively avoids the "catastrophic forgetting" problem, where training with new data causes the model to forget existing knowledge. EWC ensures the stability of parameter updates by calculating the importance of each parameter (e.g., the Fisher information matrix) and applying regularization constraints on important parameters during training.

[0103] During incremental training, corrected feedback data is integrated with historical inference results to generate an incremental training dataset. This dataset is then assigned to the corresponding submodules for training based on the model structure (e.g., the classification task module, regression task module, and causal reasoning module of a multi-task joint learning model). Training utilizes gradient descent optimization algorithms (such as Adam and RMSProp), and constrains important parameters based on the EWC strategy. During training, the model parameters are gradually updated based on the error feedback from the incremental data, generating updated model parameters.

[0104] During the performance verification phase, the updated model parameters are distributed to multiple verification nodes. These nodes can be distributed healthcare institution servers, financial risk control nodes, or industrial fault diagnosis centers. Each verification node verifies the updated model's inference performance based on a local dataset. This performance verification utilizes a variety of evaluation metrics, such as sensitivity (True Positive Rate), specificity (True Negative Rate), accuracy, and AUC (Area Under Curve). These metrics effectively measure the model's performance in different scenarios, ensuring that the updated model not only performs well on corrected feedback data but also exhibits stable performance in real-world application scenarios.

[0105] During the performance evaluation and parameter screening phase, the model's sensitivity and specificity are compared against preset thresholds, and only model parameters that meet these thresholds are retained. These filtered model parameters are then merged with the frozen parameters of the original model to generate the final updated model. Frozen parameters refer to parts of the model that are not updated (such as the feature extraction layer of a pre-trained model). This design ensures model stability and generalization capabilities.

[0106] During the new model registration phase, the updated model is registered in the model management repository as the current inference model. Model registries typically include version control mechanisms, supporting rollbacks and comparisons, ensuring a quick switch to a stable version in the event of performance degradation or anomalies. The registration information for a new model version typically includes the model version number, training time, incremental dataset identifier, performance metrics, and the encrypted storage path for the model parameter files.

[0107] For example, in the healthcare sector, correction feedback data comes from doctors' revised diagnoses. For example, an inference model might generate a diagnosis of "pneumonia, 85% probability," but a doctor, after review, may revise it to "bronchitis, 60% probability." These revisions are recorded as correction feedback data and integrated with historical diagnostic data to form an incremental training dataset. During incremental training, an elastic weight consolidation algorithm is used to ensure that the model retains existing medical knowledge during updates. After training is complete, the updated model is validated on validation nodes across multiple hospitals. Only when both sensitivity and specificity exceed preset thresholds is the model registered as a new version of the inference model.

[0108] In the financial sector, correction feedback data comes from risk adjustments made by financial analysts. For example, a credit score generated by an inference model might be "medium risk" with a default probability of "20%." However, based on client information, the analyst may revise it to "high risk, 35% default probability." This correction data is incorporated into the incremental training dataset and used to train the model. The updated model is cross-validated on validation nodes at multiple financial institutions. Only when the accuracy and AUC meet pre-set standards will it be registered as a new model version.

[0109] In the field of industrial equipment fault diagnosis, correction feedback data comes from engineers' corrections to fault classifications. For example, an inference model might predict a fault as "mechanical failure," but an engineer might correct it to "electrical failure." This correction information is integrated into an incremental training dataset and used to train the model. The updated model is then validated for fault detection on multiple industrial equipment monitoring nodes. Only when the prediction accuracy exceeds 90% is it registered as a new model version.

[0110] This embodiment can achieve dynamic adaptive optimization of the model by updating the parameters of the inference model based on the correction feedback data. The correction feedback data comes directly from the review and correction in the actual application scenario, and can reflect the errors and improvement space of the model in real applications. Incremental training based on the elastic weight consolidation algorithm can ensure that the model retains the original knowledge during the update process, avoiding catastrophic forgetting problems. The multi-node verification mechanism ensures the stability and adaptability of the updated model. Only after passing the performance evaluation of the verification node will the updated model be registered in the model management library.

[0111] The present invention relates to the field of intelligent decision-making technology and can be applied to business scenarios such as financial technology and medical health. A decision-making method, device, equipment and medium based on multimodal data are disclosed, including: collecting original data and performing cleaning and labeling processing to generate cleaned and labeled data; performing format conversion and desensitization processing on the cleaned and labeled data to generate standardized data; performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector; performing inference processing based on the multimodal feature vector through an inference model to generate inference result data; pushing the inference result data to a review terminal and receiving corrected feedback data returned by the review terminal; updating the parameters of the inference model based on the corrected feedback data to generate an updated inference model. The present invention realizes the effective fusion and precise reasoning of the multimodal features of the data by performing multimodal feature extraction and inference model inference processing after data cleaning, format conversion and desensitization processing, significantly improving the accuracy of decision-making and data security, and ensuring the privacy protection of data and dynamic model updates throughout the entire process.

[0112] In one embodiment, the above step S10 includes:

[0113] S101, acquiring real-time time series data through a sensor device, acquiring raw image data through an image acquisition device, and receiving unstructured text data through a text interface;

[0114] S102, performing sampling rate alignment processing on the real-time time series data to generate aligned time series data, performing resolution normalization processing on the original image data to generate standardized image data, and performing encoding format conversion on the unstructured text data to generate uniformly encoded text data;

[0115] S103, performing named entity recognition on the uniformly encoded text data using a pre-trained language model to generate entity objects and relationships between entities;

[0116] S104, mapping the entity object to the standardized node in the domain knowledge graph to generate structured annotation data with semantic labels;

[0117] S105, filling missing segments in the aligned time series data using a neighboring data interpolation algorithm to generate complete time series data;

[0118] S106, using a generative restoration model to complete the incomplete area in the standardized image data to generate restored image data;

[0119] S107: Integrate the complete time series data, the restored image data, and the structured annotation data to generate multimodal cleaned annotation data.

[0120] In this embodiment, collecting raw data and performing cleansing and annotation processing on it is the foundation for achieving data quality improvement and multimodal feature analysis. This covers the format standardization and quality improvement of multiple data sources and types. This process is divided into five key technical features: data collection, multimodal data preprocessing, structured annotation, data repair, and multimodal data integration. Each feature plays a different role in the data processing chain.

[0121] First, data acquisition is the starting point of data processing, involving three data sources: real-time time series data, raw image data, and unstructured text data. Real-time time series data is acquired through sensor devices, such as electrocardiogram (ECG) sensors, temperature sensors, and pressure sensors. The acquired data typically exists in the form of a time series with timestamps and numerical fields. Raw image data is acquired through image acquisition devices, such as cameras, CT scanners, MRI devices, or microscopes. Unstructured text data is received through text interfaces. This data source can include electronic medical records, contract documents, technical documents, and other sources. These data sources cover different modalities and can describe the target object or event from multiple perspectives.

[0122] After data collection is completed, the data preprocessing stage performs format conversion and alignment processing on different types of data. For real-time time series data, sampling rate alignment processing is performed, that is, time series data collected at different frequencies are adjusted to the same sampling frequency. For example, the sampling rate of electrocardiogram data is aligned from 250Hz to 500Hz to ensure the continuity and consistency of data analysis. For raw image data, resolution standardization processing is performed, that is, images of different resolutions are converted to a predefined standard resolution, such as adjusting a 1024x768 resolution image to 512x512 pixels. This processing ensures that the image has a consistent input format in subsequent model processing. For unstructured text data, encoding format conversion is performed, for example, multiple character encodings (such as GB2312, UTF-8) are uniformly converted to UTF-8 encoding format. This processing can eliminate the garbled code problem caused by incompatible character encodings.

[0123] Next, structured annotation of text data is the core of text information extraction and semantic understanding. In the process of text data structuring, named entity recognition (NER) is performed on uniformly encoded text data through pre-trained language models (such as BERT, RoBERTa). This process can automatically identify entity objects in the text (such as "patient name", "drug name", "time") and analyze the association relationship between entities (such as "drug name-treatment disease"). These entity objects and association relationships constitute the semantic labels of text information. To ensure semantic consistency, these entity objects are mapped to standardized nodes in the domain knowledge graph. For example, "hypertension" is mapped to the standard disease code "I10". This mapping can convert text information into structured data and avoid semantic ambiguity and terminology inconsistency.

[0124] In the quality restoration process of time series data and image data, the neighboring data interpolation algorithm is used to fill in the missing fragments in the time series data. Neighboring data interpolation is a numerical interpolation method based on adjacent data points, and algorithms such as linear interpolation, spline interpolation, or Gaussian interpolation can be used. For example, in electrocardiogram data, if there is missing data between time points t1 and t3, the interpolation value of t2 is calculated using the data values of t1 and t3. For image data, the restoration adopts a generative restoration model, that is, a deep convolutional neural network (such as U-Net, GPT-4Vision) is used to generate the content of the missing area in the image. This restoration method can not only fill the defects in the image, but also maintain the consistency of the style and texture of the repaired area with the original image.

[0125] Finally, multimodal data integration combines the three processed data types into a single multimodal cleaned and annotated dataset. Specifically, the complete time series data, repaired image data, and structured annotated data are aligned and stored in a multimodal format. For example, in medical applications, a patient's electrocardiogram (time series data), MRI images (image data), and electronic medical records (text data) can be combined into a single multimodal dataset for subsequent feature extraction and model training.

[0126] This embodiment can effectively improve the integrity, consistency and accuracy of the data by collecting raw data and performing cleaning and annotation processing on the raw data. By cleaning and annotating multimodal data, the semantics and format consistency between different modal data (time series, image, text) can be ensured. The use of neighboring data interpolation algorithms to fill missing fragments in time series data can avoid the impact of data gaps on subsequent analysis. The integrity of the image data can be improved by completing the missing areas in the image through the image restoration model. Through the combination of language models and knowledge graphs, the unstructured information in text data can be converted into structured semantic labels.

[0127] In one embodiment, the above step S20 includes:

[0128] S201, performing time series resampling processing on the time series data in the cleaned and labeled data to generate time series data in a unified format;

[0129] S202, performing color space conversion and tensor normalization processing on the image data in the cleaned and labeled data to generate a normalized image tensor;

[0130] S203, mapping the semantic tags in the structured annotation data in the cleaned annotation data to a predefined classification coding table to generate unified coding annotation data;

[0131] S204, adding differential privacy noise to the sensitive fields in the unified format time series data to generate desensitized time series data;

[0132] S205, performing pixel fuzzification processing on the identity-related region in the standardized image tensor to generate anonymized image data;

[0133] S206, performing hash encryption replacement on the privacy attributes in the uniformly coded annotation data to generate encrypted annotation data;

[0134] S207: Integrate the desensitized time series data, anonymized image data, and encrypted annotation data to generate multimodal standardized data.

[0135] In this embodiment, format conversion and desensitization of cleaned and annotated data are key steps in ensuring data security and format consistency, involving multiple processing operations on time series data, image data, and structured annotated data. Format conversion standardizes the data format, ensuring compatibility and data consistency in subsequent model processing; desensitization ensures data security and privacy protection. This process includes seven key technical features: time series resampling, image data standardization, semantic label mapping, differential privacy noise addition, image pixel blurring, privacy attribute hashing encryption, and multimodal data integration.

[0136] First, time series resampling is performed on the time series data to generate time series data in a unified format. Time series resampling is a processing method that rescales time series data to a fixed time interval. Resampling methods can include upsampling and downsampling. Upsampling is to add data points to the original data through interpolation methods, such as generating 1000Hz data through linear interpolation from 500Hz electrocardiogram data. Downsampling is to reduce the number of data points through data averaging or sampling, such as downsampling 1000Hz data to 100Hz. This processing ensures that the time series data has a unified time step in subsequent analysis, facilitating feature extraction and model processing.

[0137] Next, color space conversion and tensor normalization are performed on the image data to generate a standardized image tensor. Color space conversion converts the original image data from RGB or BGR color space to grayscale, YUV, or HSV color space. This conversion simplifies the image data and reduces computational complexity. Tensor normalization converts the image data into a tensor format and performs normalization. For example, image pixel values are normalized to the range of [0, 1] or [-1, 1]. Tensor normalization can also include performing mean-standard deviation normalization on the image data to conform to a standard normal distribution. This normalization process can eliminate the impact of image brightness and contrast differences on model training.

[0138] For the semantic tags in the structured annotated data, a predefined classification code table mapping is performed to generate uniformly coded annotated data. Semantic tags are typically textual information, such as "hypertension" and "diabetes." By mapping these tags to a classification code table, for example, mapping "hypertension" to "D01," data standardization can be achieved. This classification code table can be predefined based on field-specific standards (such as ICD-10 disease codes), ensuring semantic consistency and scalability of the data.

[0139] Differential privacy noise is added to sensitive fields in time series data to generate desensitized time series data. Differential privacy is a data protection technique that adds noise to data to mask personal information. For example, in a patient's heart rate data, random noise from a Gaussian or Laplace distribution is added to each data point. This noise ensures that even if a malicious user obtains the data, the patient's original data cannot be accurately restored. The amount of differential privacy noise added depends on the privacy budget ε, with a smaller ε indicating higher privacy protection.

[0140] Pixel blurring is performed on identity-associated areas in the image data to generate anonymized image data. Identity-associated areas typically include faces or other identifiable features. Pixel blurring can be performed using methods such as Gaussian blur, mosaic, or local defocus. Gaussian blurring smoothes the pixels within an area using a Gaussian kernel function, while mosaicking divides the pixels within an area into blocks and takes the average value. This process ensures that personal information in the image is not leaked.

[0141] Hash encryption is performed to replace the privacy attributes in structured annotated data, generating encrypted annotated data. Hash encryption is a one-way encryption algorithm that uses algorithms such as SHA-256 or SHA-3 to convert sensitive information (such as names and ID numbers) into a fixed-length encrypted string. The one-way nature of hash encryption ensures that even if the encrypted data is obtained, the original information cannot be reversed. This encryption method ensures the security of sensitive information during data transmission and storage.

[0142] Finally, desensitized time series data, anonymized image data, and encrypted annotated data are integrated to generate multimodal standardized data. This integration can be done through table association or tensor concatenation. For example, time series data, image data, and text data can be combined into a single multimodal data object to ensure data consistency and interoperability in subsequent model processing.

[0143] This embodiment effectively ensures data format consistency and privacy by performing format conversion and desensitization on cleaned and annotated data. Format conversion uniformly formats time series data, image data, and text data, avoiding processing errors caused by data format differences. Desensitization ensures data privacy and security through various means, including differential privacy, pixel blurring, and hash encryption, effectively reducing the risk of data leakage.

[0144] In one embodiment, the above step S30 includes:

[0145] S301, performing convolution and pooling operations on the anonymized image data in the standardized data through a three-dimensional convolutional neural network to generate an image feature vector;

[0146] S302, performing time series modeling on the desensitized time series data in the standardized data through a long short-term memory network to generate a time series feature vector;

[0147] S303, performing context encoding on the encrypted labeled data in the standardized data using a pre-trained language model to generate a text feature vector;

[0148] S304, mapping the image feature vector, the time series feature vector, and the text feature vector to the same latent space dimension to generate an aligned feature vector;

[0149] S305, determining the interaction weight of the aligned feature vectors through a cross-modal attention mechanism to generate a weighted fusion feature vector;

[0150] S306 , performing dimensionality reduction and normalization processing on the weighted fusion feature vector to generate a unified representation multimodal feature vector.

[0151] In this example, different neural network structures are used to extract features from image data, time series data, and text data. These features are then aligned in the same latent space and fused through a cross-modal attention mechanism to ultimately generate a unified multimodal feature vector. This process ensures the effective integration of information from multiple data sources, improving the model's perception and reasoning capabilities.

[0152] First, a three-dimensional convolutional neural network (3D-CNN) is used to perform convolution and pooling operations on the anonymized image data in the standardized data to generate an image feature vector. 3D-CNN is a deep neural network structure specifically designed to process three-dimensional data or data with multi-channel features. The convolution operation extracts spatial features by sliding the convolution kernel across the image; the pooling operation reduces the size of the feature map through maximum pooling or average pooling to retain important features. For medical images (such as MRI or CT), 3D-CNN can simultaneously capture spatial structural information (such as three-dimensional anatomical structure) and visual features. The size and stride of the convolution kernel can be dynamically adjusted, such as a 3×3×3 convolution kernel and a 2×2×2 pooling kernel, to achieve a balance between resolution and computational efficiency. The image feature vector is a high-dimensional feature representation that can effectively capture the structural information of image data.

[0153] Secondly, a long short-term memory (LSTM) network is used to perform time series modeling on the desensitized time series data in the standardized data to generate a time series feature vector. LSTM is a special type of recurrent neural network (RNN) that solves the vanishing gradient problem in traditional RNNs by introducing forget gates, input gates, and output gates. LSTM can effectively learn long-term dependencies in time series data and is particularly effective for processing time series data such as electrocardiograms, respiratory rate, and blood sugar changes. In actual implementation, the number of LSTM units can be adjusted based on the data length and time step size, for example, 128 LSTM units can process 1 minute of electrocardiogram data. The time series feature vector is a dynamic feature representation that can capture the patterns of data changes over time.

[0154] Context encoding is performed on the encrypted annotated data in the standardized data through a pre-trained language model (such as BERT, GPT) to generate a text feature vector. A pre-trained language model is a deep neural network trained with large-scale corpus that can automatically learn the semantic and syntactic structures in the text. Context encoding refers to generating a semantic vector representation for each text unit (such as a word or phrase) based on a given context. For structured text (such as electronic medical records and diagnostic reports), the pre-trained language model can extract key semantic information from the text. For example, "hypertension" is encoded as a vector with specific semantics. The text feature vector is a high-dimensional semantic representation that can capture the semantic information in text data.

[0155] Next, the image feature vector, time series feature vector, and text feature vector are mapped to the same latent space dimension to generate an aligned feature vector. This step is the key to multimodal feature alignment, ensuring that data from different modalities can be compared and combined in the same dimension. Feature alignment can be achieved through a fully connected layer, where each feature vector is converted to the same vector dimension, such as 512 or 1024. The feature alignment process can also be further stabilized through batch normalization or layer normalization.

[0156] Subsequently, the interaction weights of the aligned feature vectors are determined through the cross-modal attention mechanism to generate a weighted fused feature vector. The cross-modal attention mechanism is a mechanism that determines the importance of each modal feature during fusion by using attention weights. For example, if text information (such as a diagnostic report) is more important than image information in a medical scenario, the attention mechanism will automatically assign a higher weight to the text feature vector. The cross-modal attention mechanism can be implemented through dot-product attention (Dot-ProductAttention) or self-attention (Self-Attention), in which each modal feature vector calculates an attention score based on its similarity with other modal features. Finally, all feature vectors are weighted and summed according to the attention score to generate a weighted fused feature vector.

[0157] Finally, the weighted fused feature vector is subjected to dimensionality reduction and normalization to generate a unified multimodal feature vector. Dimensionality reduction can be performed using principal component analysis (PCA), autoencoders, or fully connected layers in convolutional neural networks. The goal is to reduce the dimensionality of the feature vector to reduce computational complexity. Normalization ensures that the numerical range of the feature vector remains consistent, typically using L2 regularization (normalizing the vector length to 1). The unified multimodal feature vector is a fixed-dimensional vector that can efficiently represent information from multiple data sources and serve as input to subsequent inference models.

[0158] This embodiment performs multimodal feature extraction and fusion processing on standardized data, which can effectively integrate information from three data sources: images, time series, and text in the same feature space, ensuring that the diversity and relevance of the data are fully utilized. The three-dimensional convolutional neural network (3D-CNN) effectively extracts the spatial structure information of the image, the long short-term memory network (LSTM) captures the dynamic changes of time series data, and the pre-trained language model extracts semantic information from the text. The cross-modal attention mechanism automatically assigns weights according to the importance of each modal feature to ensure the flexibility of multimodal information fusion. The multimodal feature vector finally generated has a unified feature representation, which can improve the generalization ability and prediction accuracy of the model.

[0159] In one embodiment, the above step S40 includes:

[0160] S401, calling a pre-trained multi-task joint learning model, wherein the multi-task joint learning model includes a classification task module, a regression task module, and a causal reasoning module;

[0161] S402, inputting the multimodal feature vector into the multi-task joint learning model, outputting a category label through the classification task module, outputting a continuous prediction value through the regression task module, and outputting a causal relationship path through the causal reasoning module;

[0162] S403, performing logical correction on the category labels and continuous prediction values based on the causal relationship path to generate a causal correction reasoning result;

[0163] S404, matching and verifying the causal correction reasoning result with the association strategy in the domain knowledge graph to generate a verified reasoning result;

[0164] S405 , integrating the category labels, continuous prediction values, and causal relationship paths in the verified reasoning results to generate reasoning result data.

[0165] In this embodiment, a multi-task joint learning model is used to map multimodal feature vectors into specific reasoning results, and causal reasoning and knowledge graphs are combined to enhance the interpretability and accuracy of the model.

[0166] First, call the pre-trained multi-task joint learning model, which is a neural network framework that integrates multiple tasks and can complete multiple prediction tasks in a single reasoning process. Specifically, the multi-task joint learning model includes a classification task module, a regression task module and a causal reasoning module. The classification task module is a deep neural network (such as a fully connected neural network or a convolutional neural network) that can output category labels based on the input feature vector. For example, in the medical field, it can output disease diagnosis labels (such as "pneumonia" or "hypertension"), and in the financial field, it can output customer risk levels (such as "low risk" and "high risk"). The regression task module is usually based on a fully connected network or a multi-layer perceptron (MLP) and can output continuous prediction values, such as predicting blood sugar levels, heart rate or stock prices. The causal reasoning module determines the causal relationship between the input features by constructing a causal graph or using a causal Bayesian network, and outputs the causal relationship path.

[0167] Secondly, the multimodal feature vector is input into the multi-task joint learning model. This feature vector is a unified representation vector generated by the previous step, which contains information from image, time series and text data. The multi-task joint learning model automatically assigns the feature vector to the corresponding task module according to the task type. The classification task module outputs a category label, which represents the classification result, such as "heart disease" or "normal". The regression task module outputs a continuous prediction value, such as the heart rate prediction value of "75 beats / minute" and the blood glucose prediction value of "5.8mmol / L". The causal reasoning module outputs a causal relationship path, which represents the causal relationship between each feature, such as "hypertension → heart disease".

[0168] Subsequently, the category labels and continuous prediction values are logically corrected based on the causal relationship path to generate a causal correction reasoning result. This process uses the causal path generated in the causal reasoning module to verify and correct the consistency of the reasoning results. Specifically, if the causal path indicates that "high blood pressure" is a causal factor of "heart disease", then if the classification task module does not predict "heart disease", it needs to be corrected to "heart disease" based on the causal relationship. Logical correction methods can include rule correction (based on predefined rules) and dynamic correction (based on adaptive adjustment) to ensure that the reasoning results are consistent with the causal reasoning logic.

[0169] Next, the causal correction reasoning results are matched and verified with the association strategies in the domain knowledge graph to generate the verified reasoning results. The knowledge graph is a semantic network that represents domain knowledge as a graph structure, where nodes represent domain concepts (such as diseases, symptoms, and drugs) and edges represent the associations between them (such as "treatment relationships" and "symptom manifestations"). During the reasoning process, the causal correction reasoning results will be matched with the rules or association strategies stored in the knowledge graph. If the reasoning results do not match the medical rules or financial rules already in the knowledge graph, they will be corrected or marked as abnormal. The verified reasoning results ensure that the reasoning results meet the constraints of domain knowledge and improve accuracy.

[0170] Finally, the category labels, continuous prediction values, and causal relationship paths in the verified inference results are integrated to generate structured inference result data. Structured inference result data is a standardized data format that can record classification results, prediction values, and causal relationships in the form of structured fields, facilitating subsequent review, display, and further analysis. Specifically, structured inference result data can be formatted in JSON or XML and contain the following fields:

[0171] Classification labels: such as "heart disease" and "normal";

[0172] Predicted value: such as "blood sugar level 5.8mmol / L";

[0173] Causal path: such as "high blood pressure → heart disease";

[0174] Verification status: such as "Passed verification" or "Failed verification".

[0175] This embodiment performs reasoning processing based on multimodal feature vectors through an inference model, which can effectively integrate information in multimodal data and generate accurate reasoning results. The multi-task joint learning model realizes the simultaneous generation of multiple reasoning results through classification tasks, regression tasks and causal reasoning tasks. The causal reasoning module combines the causal relationship path to enhance the interpretability and logic of the reasoning results, ensuring that the reasoning results are not just the results of data statistics, but reasonable inferences based on the causal relationship of the data. Domain knowledge graph verification further improves the accuracy and reliability of the reasoning results and avoids reasoning results that violate domain knowledge. The structured reasoning result data finally generated has a standardized output format, which is convenient for storage, transmission and subsequent analysis.

[0176] In one embodiment, the above step S60 includes:

[0177] S601, performing deduplication and label alignment processing on the corrected feedback data and the historical reasoning result data to generate an incremental training data set;

[0178] S602, partially unfreeze the parameters of the inference model using an elastic weight consolidation algorithm, and update the unfrozen parameters based on the incremental training data set to generate updated model parameters;

[0179] S603, distributing the updated model parameters to multiple verification nodes, performing reasoning performance verification based on the local data set of each verification node, and generating verification result indicators;

[0180] S604, determining the sensitivity and specificity of the model based on the validation result indicators, and screening out updated model parameters that meet a preset threshold;

[0181] S605: Based on the filtered updated model parameters and frozen parameters, a new version of the inference model is constructed and the model registry is updated.

[0182] In this embodiment, the parameters of the inference model are updated based on the corrected feedback data to generate an updated inference model. This process achieves adaptive optimization of the inference model through an incremental learning mechanism. The entire process includes incremental dataset construction, model parameter optimization, multi-node performance verification, and model version management.

[0183] First, the correction feedback data and the historical reasoning result data are deduplicated and label aligned to generate an incremental training dataset. The correction feedback data is the correction information obtained from the review terminal or the automatic review module, which contains the corrected category labels, predicted values or causal relationship paths. These correction data represent the errors or inaccurate reasoning results found in actual applications. The historical reasoning result data is the prediction data generated by the reasoning model during the historical reasoning process. The deduplication operation is to eliminate duplicate data and ensure that each data in the incremental training dataset is independent, so as to avoid data redundancy affecting the model training effect. Label alignment is to ensure that the label format in the correction feedback data is consistent with the historical data, such as converting the corrected category label into a standardized format (such as from "hypertension" to "Hypertension"). The final generated incremental training dataset includes the corrected multimodal features and corrected labels to ensure that the model training can absorb the latest correction information.

[0184] Secondly, the Elastic Weight Consolidation (EWC) algorithm is used to partially unfreeze the inference model parameters. These unfrozen parameters are then updated based on the incremental training dataset to generate updated model parameters. EWC is an incremental learning method designed to avoid catastrophic forgetting. Specifically, it preserves the model's existing knowledge while allowing new knowledge to be learned during incremental learning. The specific implementation is as follows: First, some parameters of the inference model (such as weights and biases) are unfrozen, allowing these parameters to be updated during incremental training. Gradient descent optimization is then performed on the unfrozen parameters based on the incremental training dataset to update the model parameters. EWC introduces a regularization term to limit the update range and ensure that knowledge from the original task is not forgotten. The regularization term is calculated based on the Fisher Information Matrix, which measures the importance of each parameter to the model output. Parameters with high importance are subject to stricter constraints during training, while parameters with low importance can be adjusted more freely. Ultimately, the updated model parameters reflect the knowledge gained from the corrected feedback data while maintaining the inference performance of the original model.

[0185] The updated model parameters are then distributed to multiple validation nodes, and inference performance is verified based on each validation node's local dataset, generating validation result metrics. Validation nodes are independent computing nodes deployed in different environments or regions. For example, in a medical scenario, validation nodes could be data centers in multiple hospitals, each using a local patient dataset for model validation. In a financial scenario, validation nodes could be local data from different branches. After the updated model parameters are distributed to each validation node, the model performs inference on the local dataset and generates inference results. These inference results are compared with the true labels in the local dataset to calculate performance metrics such as accuracy, sensitivity, and specificity. These performance metrics reflect the model's generalization performance across different environments and data distributions, ensuring that the model not only performs well on the training data but also exhibits stable performance in real-world application environments.

[0186] Next, the model sensitivity and specificity are determined based on the validation results, and updated model parameters that meet the preset thresholds are selected. Sensitivity refers to the proportion of positive samples correctly identified by the model, while specificity refers to the proportion of negative samples correctly identified by the model. These two metrics are core indicators for evaluating the performance of classification models. In practical applications, thresholds for sensitivity and specificity can be set according to business needs. For example, in medical diagnosis, high sensitivity may be prioritized (to reduce missed diagnoses), while in the financial field, high specificity (to reduce misjudgments) may be more important. By performing statistical analysis on the performance indicators of all validation nodes, updated model parameters that meet the preset thresholds are selected.

[0187] Finally, based on the selected updated model parameters and frozen parameters, a new version of the inference model is constructed and the model registry is updated. Frozen parameters are model parameters that were not updated during the elastic weight consolidation process; these parameters maintain the model's original inference capabilities. The new version of the inference model is composed of both frozen and updated parameters, ensuring that the new model retains its original inference capabilities while absorbing new knowledge. The constructed new version of the inference model is stored in the model registry, a version management system that records the parameter configuration, training dataset, and performance metrics of each model version, ensuring traceability and reproducibility of model versions.

[0188] This embodiment updates the parameters of the inference model by correcting the feedback data, thereby achieving continuous adaptive optimization of the model. An incremental training data set is constructed based on the corrected feedback data and historical inference result data to ensure that the model can absorb the latest correction information in a timely manner. The elastic weight consolidation algorithm avoids the problem of catastrophic forgetting of the model during incremental training, allowing the model to maintain its original reasoning ability while learning new knowledge. The multi-node verification mechanism ensures the generalization performance of the model in different application environments through distributed verification, and the setting of sensitivity and specificity thresholds ensures that the model performance meets the actual application requirements. Finally, the model version is managed through the model registry, giving the model version traceability and version control capabilities.

[0189] In one embodiment, after the above step S60, the method further includes:

[0190] S701, performing symmetric encryption algorithm processing on the desensitized time series data, anonymized image data, and encrypted annotation data in the standardized data to generate encrypted standardized data;

[0191] S702, performing homomorphic encryption processing on the multimodal feature vector to generate an encrypted multimodal feature vector;

[0192] S703, performing hash desensitization processing on the privacy attribute field in the inference result data to generate a hash desensitized inference result;

[0193] S704, performing dynamic mask processing on the sensitive information in the corrected feedback data to generate dynamic mask feedback data;

[0194] S705, performing key sharding encryption processing on the parameter file of the updated inference model to generate encrypted model parameters;

[0195] S706, deploying an access control policy on the data access interface and recording access requests to the encrypted standardized data, encrypted multimodal feature vectors, hash desensitization inference results, dynamic mask feedback data, and encryption model parameters;

[0196] S707, generating an audit log with the operation type, timestamp, and user identity information of the access request;

[0197] S708, performing hash processing and distributed evidence storage on the audit log through blockchain technology to generate an audit tracking record that cannot be tampered with.

[0198] In this embodiment, after updating the inference model parameters based on the corrected feedback data, privacy protection is applied to the data and model to ensure data security and access audit integrity, and a traceable audit trail is generated. This process encompasses data encryption, access control, audit record generation, and distributed evidence storage based on blockchain.

[0199] First, symmetric encryption algorithms are applied to the desensitized time series data, anonymized image data, and encrypted annotation data in the standardized data to generate encrypted standardized data. Symmetric encryption is an encryption method that uses the same key for both encryption and decryption. Desensitized time series data is achieved using techniques such as differential privacy or noise addition, while anonymized image data is achieved through techniques such as pixel blurring, camouflage, or generative repair. The encrypted annotation data is generated by encrypting the private attribute fields in the structured annotation data using hashing or other irreversible encryption algorithms. Symmetric encryption algorithms can use the Advanced Encryption Standard (AES-256), whose security relies on the confidentiality of the key. In the specific implementation, data is encrypted in blocks, with the block size determined by the data type (e.g., fixed-length blocks for time series data and pixel matrices for image data). Keys are dynamically generated by the key management service and securely distributed to the data encryption module.

[0200] Next, homomorphic encryption is performed on the multimodal feature vector to generate an encrypted multimodal feature vector. Homomorphic encryption is an encryption method that allows specific operations to be performed on encrypted data, that is, operations such as addition and multiplication are performed without decryption. The multimodal feature vector here is extracted and fused from standardized data, including image features, time series features, and text features. The use of homomorphic encryption can ensure that even if the feature vector is transmitted or stored in an encrypted state, the inference model can still perform operations on the encrypted data. For example, through Paillier encryption or CKKS (Cheon-Kim-Kim-Song) encryption scheme, matrix multiplication and weighted summation of encrypted feature vectors can be performed in the inference model, thereby realizing inference processing in an encrypted state. The encrypted multimodal feature vector always remains encrypted during transmission and storage, and is only decrypted by the authorized entity using a private key when decryption is required.

[0201] Then, hash desensitization is performed on the privacy attribute fields in the inference result data to generate a hash-desensitized inference result. Privacy attribute fields refer to sensitive information that may be contained in the inference result data, such as patient identity information in medical diagnosis results or customer identity information in financial credit scores. Hash desensitization converts sensitive information into a hash value using an irreversible hash function (such as SHA-256), ensuring that the sensitive information cannot be restored. Hash desensitization not only protects privacy during data transmission and storage, but also enables identity protection in data analysis and statistics.

[0202] Furthermore, dynamic masking is performed on the sensitive information in the correction feedback data to generate dynamic mask feedback data. Sensitive information includes user identification, correction content, or other private data involved in the correction feedback data. Dynamic masking is a technology that replaces sensitive information by generating a mask (such as random symbols or asterisks) in real time. The dynamic nature of the mask is reflected in the dynamic adjustment of the mask style and display range based on user permissions or data sensitivity. For example, for highly sensitive data, a full mask (such as "*****") can be used, while for low-sensitivity data, a partial mask (such as "123****") can be used. Dynamic masking technology ensures that sensitive information is protected both during display and storage.

[0203] In addition, key sharding encryption is performed on the parameter file of the updated inference model to generate encrypted model parameters. Key sharding encryption is a technology that divides the encryption key into multiple shards, each of which is separately encrypted and distributed. Even if an attacker obtains one or more of the shards, the model parameters cannot be decrypted. In the specific implementation, the model parameter file is first encrypted (such as AES-256 encryption), and then the key is divided into multiple shards (such as based on Shamir's SecretSharing), and each shard is distributed to a different key storage node. When recovering the model parameters, it is necessary to combine the keys from at least a certain number of shards (such as 3 / 5 shards).

[0204] Deploy access control policies on data access interfaces and record access requests for encrypted standardized data, encrypted multimodal feature vectors, hashed desensitized inference results, dynamically masked feedback data, and encrypted model parameters. Access control policies include authentication (such as OAuth 2.0), permission verification (such as role-based access control (RBAC), and operation records (such as access logs). Each access request (such as reading, modifying, or deleting data) is recorded, including the operation type (such as read, modify), timestamp, and user identity information. Access control ensures that only authorized users can access sensitive data or models.

[0205] Ultimately, an audit log is generated, including the access request's operation type, timestamp, and user identity information. This log is then hashed and distributed using blockchain technology, creating an unalterable audit trail. Blockchain technology is an immutable data recording technology based on a distributed ledger and consensus mechanism. Each audit log entry is hashed after being recorded. The hash value, along with the timestamp and user identity information, is then combined to create a blockchain transaction, which is stored in the distributed ledger. Due to the blockchain's immutability, even if an attacker attempts to modify or delete the log, they cannot tamper with the record without being detected.

[0206] Example: In the healthcare sector, a comprehensive hospital built an intelligent diagnostic assistance system. First, the hospital deployed a data acquisition module. Sensors collected patients' physiological monitoring data (such as heart rate, blood pressure, and blood oxygen saturation) in real time. Medical imaging equipment acquired CT and MRI images of patients. Unstructured text data (such as physicians' diagnostic reports and laboratory test reports) was received from the electronic medical record system via a text interface. This raw data was automatically uploaded to the data processing platform, which first cleaned and annotated the collected data. Time series data was aligned using sampling rates to eliminate sampling variations between different monitoring devices. Image data was normalized to a uniform resolution to ensure consistency during analysis. Text data was converted to a unified encoding format, and a pre-trained language model was used to identify medical named entities (such as drug names, symptoms, and diseases) to generate structured annotated data. Missing segments in the time series data were automatically filled using a neighboring data interpolation algorithm, while incomplete regions in the image data were automatically completed using a generative repair model. Ultimately, the complete time series data, repaired image data, and structured text data were integrated to form a multimodal cleaned and annotated dataset. After data cleaning and labeling, the system further performs format conversion and desensitization on the cleaned and labeled data. Specifically, time series data is resampled to a unified format, image data is converted to a standard color space, and processed into a standardized tensor format. Semantic labels in structured text data are mapped to a predefined classification coding table to ensure label standardization. Differential privacy noise is added to sensitive fields in the time series data (such as patient ID and medical record number) to ensure privacy protection. Identity-related areas in image data (such as patient facial features) are anonymized through pixel blurring. Private attributes in structured text data (such as patient name and address) are replaced with hash encryption to ensure irreversible privacy restoration. Finally, this desensitized and encrypted data is integrated into a multimodal standardized dataset. Based on this multimodal standardized dataset, the system performs multimodal feature extraction and fusion on different data types. Medical image data uses a three-dimensional convolutional neural network to extract image features, while time series data uses a long short-term memory network to perform time series modeling and extract temporal features. Structured text data uses a pre-trained language model (such as BERT or GPT) to perform contextual encoding and extract text features. All features are mapped to the same latent space and interact with each other through a cross-modal attention mechanism to generate a weighted fused feature vector. Finally, through dimensionality reduction and normalization, a unified multimodal feature vector is generated. During the inference phase, the system calls a pre-trained multi-task joint learning model, which includes a classification task module, a regression task module, and a causal reasoning module. After the multimodal feature vector is input into the model, the classification module outputs the patient's possible disease category label, the regression module outputs the disease risk probability, and the causal reasoning module determines the causal relationship between diseases based on the domain knowledge graph.Based on the causal relationship paths generated by the causal inference module, the system logically corrects the initial prediction results (category labels and risk probabilities) to ensure their medical plausibility. The system then verifies the corrected prediction results by matching them with the medical rules in the domain knowledge graph, generating validated inference results and integrating them into structured inference result data. The inference result data is automatically pushed to the physician's terminal, where the physician can modify the inference results based on their own experience, such as revising disease labels, adjusting risk probabilities, or supplementing diagnostic evidence. The physician's feedback on these corrections is recorded and fed back to the system in real time. Based on these corrections, the system automatically performs incremental learning to update the parameters of the inference model. First, the corrected feedback data is deduplicated and label-aligned with historical inference result data to generate an incremental training dataset. An elastic weight consolidation algorithm is used to unfreeze some parameters of the inference model. Incremental training is then performed on the incremental training dataset to optimize the unfrozen parameters. The updated model parameters are distributed to multiple hospital nodes for performance verification based on each node's local dataset. Based on the verification results, the system determines the model's sensitivity and specificity and selects updated model parameters that meet preset thresholds. Finally, the system builds a new version of the inference model and updates it to the model registry. To ensure data security and privacy, the system further implements encryption and audit protection for the data and model. The desensitized time series data, anonymized image data, and encrypted annotation data in the standardized data are symmetrically encrypted (e.g., AES-256 encryption). Multimodal feature vectors are homomorphically encrypted, ensuring that calculations can be completed without decryption during inference. Private fields in the inference result data (e.g., patient name and ID) are hashed and desensitized. Sensitive information in the correction feedback data (e.g., doctor's signature) is dynamically masked. The parameter file of the updated inference model is encrypted with a sharded key, ensuring that even if part of the key is leaked, the model parameters cannot be recovered. The system implements access control policies on the data access interface and automatically logs all access requests, including access operations to encrypted standardized data, encrypted multimodal feature vectors, hashed desensitized inference results, dynamically masked feedback data, and encrypted model parameters. All access logs are generated into audit logs in real time and stored using blockchain technology to ensure the immutability of the logs and the traceability of access behavior.

[0207] In the financial sector, a bank is building an intelligent credit risk control system. First, the bank collects raw data from customer transaction records, customer identification images, and credit report texts submitted by customers. Transaction records are standardized into time series data, and missing data is filled using interpolation algorithms. Customer images are standardized and anonymized using fuzzification techniques. A pre-trained language model is used to identify basic customer information (such as name, occupation, and income) from the credit report text, generating structured annotated data. All cleaned and annotated data is converted into a standardized dataset format and privacy-preserving features are applied. In the feature extraction phase, customer transaction records are fed into a long-short-term memory network to generate time series feature vectors, customer images are fed into a convolutional neural network to generate image feature vectors, and text feature vectors are extracted from the credit report text using a language model. All features are aligned into the same latent space and fused using a cross-modal attention mechanism to generate a unified multimodal feature vector. In the inference phase, the bank's intelligent credit model performs predictions based on a multi-task joint learning model. The classification module outputs the customer's credit rating, the regression module outputs the customer's credit score, and the causal inference module generates causal explanation paths based on the domain knowledge graph. The system automatically verifies the inference results and generates structured inference result data. Credit managers can revise their inference results based on their experience and feed this revised feedback back into the system. Based on this revised feedback, the bank's intelligent credit model performs incremental learning and updates the model. This revised data is aligned with historical inference results to generate an incremental training dataset. The system updates model parameters using an elastic weight consolidation algorithm and builds a new version of the model after multi-node verification. The updated model and inference result data are stored in encrypted form, and all access operations are recorded and documented using blockchain technology to ensure data security and compliance.

[0208] This embodiment achieves data security and compliance through data encryption, privacy protection, and audit tracking mechanisms. Standardized data, multimodal feature vectors, inference result data, and correction feedback data are encrypted or masked to ensure data is not leaked or tampered with during transmission and storage. Based on access control and audit logging, all data access operations are recorded, and blockchain technology ensures the immutability of audit records, ensuring secure and transparent data access. Dynamic masking and hash desensitization further enhance the privacy protection of sensitive information.

[0209] In one embodiment, a decision-making device based on multimodal data is provided, and the decision-making device based on multimodal data corresponds one-to-one with the decision-making method based on multimodal data in the above embodiment. Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of a decision-making device based on multimodal data according to the present invention. These modules include a data preprocessing module 10, a data normalization module 20, a multimodal feature extraction module 30, an inference model module 40, a feedback collection module 50, and a model update module 60. Each functional module is described in detail below:

[0210] The data preprocessing module 10 is used to collect raw multimodal data and perform cleaning and labeling processing on the raw multimodal data to obtain cleaned and labeled data;

[0211] A data standardization module 20 is used to perform format conversion and desensitization processing on the cleaned and annotated data to generate standardized data;

[0212] A multimodal feature extraction module 30 is used to perform multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector;

[0213] An inference model module 40 is configured to perform inference processing based on the multimodal feature vector through an inference model to generate inference result data;

[0214] Feedback collection module 50, used to push the inference result data to the review terminal and receive the correction feedback data returned by the review terminal;

[0215] The model updating module 60 is configured to update the parameters of the inference model based on the corrected feedback data to generate an updated inference model.

[0216] In one embodiment, the data preprocessing module 10 is specifically configured to:

[0217] Acquire real-time time series data through sensor devices, acquire raw image data through image acquisition devices, and receive unstructured text data through text interfaces;

[0218] Performing sampling rate alignment processing on the real-time time series data to generate aligned time series data, performing resolution normalization processing on the original image data to generate standardized image data, and performing encoding format conversion on the unstructured text data to generate unified encoding text data;

[0219] Performing named entity recognition on the uniformly encoded text data using a pre-trained language model to generate entity objects and relationships between entities;

[0220] Mapping the entity objects to standardized nodes in the domain knowledge graph to generate structured annotated data with semantic labels;

[0221] Filling missing segments in the aligned time series data with a neighboring data interpolation algorithm to generate complete time series data;

[0222] Completing the incomplete area in the standardized image data using a generative restoration model to generate restored image data;

[0223] The complete time series data, the repaired image data and the structured annotation data are integrated to generate multimodal cleaned annotation data.

[0224] In one embodiment, the data standardization module 20 is specifically configured to:

[0225] Performing time series resampling processing on the time series data in the cleaned and labeled data to generate time series data in a unified format;

[0226] Performing color space conversion and tensor normalization processing on the image data in the cleaned and labeled data to generate a normalized image tensor;

[0227] Mapping semantic tags in the structured annotation data in the cleaned annotation data to a predefined classification coding table to generate unified coded annotation data;

[0228] Adding differential privacy noise to sensitive fields in the unified format time series data to generate desensitized time series data;

[0229] Performing pixel blurring processing on the identity-associated region in the standardized image tensor to generate anonymized image data;

[0230] Performing hash encryption replacement on the privacy attributes in the uniformly coded annotation data to generate encrypted annotation data;

[0231] The desensitized time series data, anonymized image data and encrypted annotation data are integrated to generate multimodal standardized data.

[0232] In one embodiment, the multimodal feature extraction module 30 is specifically configured to:

[0233] Performing convolution and pooling operations on the anonymized image data in the standardized data through a three-dimensional convolutional neural network to generate an image feature vector;

[0234] Performing time series modeling on the desensitized time series data in the standardized data through a long short-term memory network to generate a time series feature vector;

[0235] Performing context encoding on the encrypted labeled data in the standardized data using a pre-trained language model to generate a text feature vector;

[0236] Mapping the image feature vector, the time series feature vector, and the text feature vector to the same latent space dimension to generate an aligned feature vector;

[0237] Determining the interaction weights of the aligned feature vectors through a cross-modal attention mechanism to generate a weighted fusion feature vector;

[0238] Dimensionality reduction and normalization are performed on the weighted fusion feature vector to generate a multimodal feature vector with unified representation.

[0239] In one embodiment, the inference model module 40 is specifically configured to:

[0240] Calling a pre-trained multi-task joint learning model, wherein the multi-task joint learning model includes a classification task module, a regression task module, and a causal reasoning module;

[0241] Inputting the multimodal feature vector into the multi-task joint learning model, outputting a category label through the classification task module, outputting a continuous prediction value through the regression task module, and outputting a causal relationship path through the causal reasoning module;

[0242] Performing logical correction on the category labels and continuous prediction values based on the causal relationship path to generate a causal correction reasoning result;

[0243] Match and verify the causal correction reasoning result with the association strategy in the domain knowledge graph to generate a verified reasoning result;

[0244] Integrate the category labels, continuous prediction values, and causal relationship paths in the verified inference results to generate inference result data.

[0245] In one embodiment, the model updating module 60 is specifically configured to:

[0246] De-duplication and label alignment are performed on the corrected feedback data and the historical reasoning result data to generate an incremental training data set;

[0247] Partially unfreeze the parameters of the inference model using an elastic weight consolidation algorithm, and update the unfrozen parameters based on the incremental training data set to generate updated model parameters;

[0248] Distributing the updated model parameters to multiple verification nodes, performing reasoning performance verification based on the local data set of each verification node, and generating verification result indicators;

[0249] Determine the sensitivity and specificity of the model based on the validation result indicators, and select updated model parameters that meet the preset thresholds;

[0250] Based on the filtered updated model parameters and frozen parameters, a new version of the inference model is built and the model registry is updated.

[0251] In one embodiment, the model updating module 60 is specifically configured to:

[0252] Performing symmetric encryption algorithm processing on the desensitized time series data, anonymized image data and encrypted annotation data in the standardized data to generate encrypted standardized data;

[0253] Performing homomorphic encryption on the multimodal feature vector to generate an encrypted multimodal feature vector;

[0254] Performing hash desensitization processing on the privacy attribute field in the inference result data to generate a hash desensitized inference result;

[0255] Performing dynamic masking processing on the sensitive information in the corrected feedback data to generate dynamic masked feedback data;

[0256] Performing key sharding encryption processing on the parameter file of the updated inference model to generate encrypted model parameters;

[0257] Deploy access control policies on data access interfaces and record access requests to the encrypted standardized data, encrypted multimodal feature vectors, hash desensitized inference results, dynamic mask feedback data, and encryption model parameters;

[0258] Generate an audit log of the operation type, timestamp, and user identity information of the access request;

[0259] The audit log is hashed and distributedly stored using blockchain technology to generate tamper-proof audit trail records.

[0260] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a decision-making method based on multimodal data on the server side.

[0261] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a decision-making method based on multimodal data.

[0262] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0263] Collecting original multimodal data, and performing cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data;

[0264] Performing format conversion and desensitization processing on the cleaned and annotated data to generate standardized data;

[0265] Performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector;

[0266] Performing reasoning processing based on the multimodal feature vector through an inference model to generate inference result data;

[0267] Pushing the inference result data to a review terminal and receiving correction feedback data returned by the review terminal;

[0268] The parameters of the inference model are updated based on the corrected feedback data to generate an updated inference model.

[0269] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0270] Collecting original multimodal data, and performing cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data;

[0271] Performing format conversion and desensitization processing on the cleaned and annotated data to generate standardized data;

[0272] Performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector;

[0273] Performing reasoning processing based on the multimodal feature vector through an inference model to generate inference result data;

[0274] Pushing the inference result data to a review terminal and receiving correction feedback data returned by the review terminal;

[0275] The parameters of the inference model are updated based on the corrected feedback data to generate an updated inference model.

[0276] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0277] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0278] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0279] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A decision-making method based on multimodal data, characterized in that: The following steps are involved: Collecting original multimodal data, and performing cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data; Performing format conversion and desensitization processing on the cleaned and annotated data to generate standardized data; Performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector; Performing reasoning processing based on the multimodal feature vector through an inference model to generate inference result data; Pushing the inference result data to a review terminal and receiving correction feedback data returned by the review terminal; The parameters of the inference model are updated based on the corrected feedback data to generate an updated inference model.

2. The decision-making method based on multimodal data according to claim 1, characterized in that: Collecting original multimodal data and performing cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data, including: Acquire real-time time series data through sensor devices, acquire raw image data through image acquisition devices, and receive unstructured text data through text interfaces; Performing sampling rate alignment processing on the real-time time series data to generate aligned time series data, performing resolution normalization processing on the original image data to generate standardized image data, and performing encoding format conversion on the unstructured text data to generate unified encoding text data; Performing named entity recognition on the uniformly encoded text data using a pre-trained language model to generate entity objects and relationships between entities; Mapping the entity objects to standardized nodes in the domain knowledge graph to generate structured annotated data with semantic labels; Filling missing segments in the aligned time series data with a neighboring data interpolation algorithm to generate complete time series data; Completing the incomplete area in the standardized image data using a generative restoration model to generate restored image data; The complete time series data, the repaired image data and the structured annotation data are integrated to generate multimodal cleaned annotation data.

3. The decision-making method based on multimodal data according to claim 1, characterized in that: Performing format conversion and desensitization processing on the cleaned and annotated data to generate standardized data, including: Performing time series resampling processing on the time series data in the cleaned and labeled data to generate time series data in a unified format; Performing color space conversion and tensor normalization processing on the image data in the cleaned and labeled data to generate a normalized image tensor; Mapping semantic tags in the structured annotation data in the cleaned annotation data to a predefined classification coding table to generate unified coded annotation data; Adding differential privacy noise to sensitive fields in the unified format time series data to generate desensitized time series data; Performing pixel blurring processing on the identity-associated region in the standardized image tensor to generate anonymized image data; Performing hash encryption replacement on the privacy attributes in the uniformly coded annotation data to generate encrypted annotation data; The desensitized time series data, anonymized image data and encrypted annotation data are integrated to generate multimodal standardized data.

4. The decision-making method based on multimodal data according to claim 1, characterized in that: Performing multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector includes: Performing convolution and pooling operations on the anonymized image data in the standardized data through a three-dimensional convolutional neural network to generate an image feature vector; Performing time series modeling on the desensitized time series data in the standardized data through a long short-term memory network to generate a time series feature vector; Performing context encoding on the encrypted labeled data in the standardized data using a pre-trained language model to generate a text feature vector; Mapping the image feature vector, the time series feature vector, and the text feature vector to the same latent space dimension to generate an aligned feature vector; Determining the interaction weights of the aligned feature vectors through a cross-modal attention mechanism to generate a weighted fusion feature vector; Dimensionality reduction and normalization are performed on the weighted fusion feature vector to generate a multimodal feature vector with unified representation.

5. The decision-making method based on multimodal data according to claim 1, characterized in that: Performing reasoning processing based on the multimodal feature vector through an inference model to generate inference result data includes: Calling a pre-trained multi-task joint learning model, wherein the multi-task joint learning model includes a classification task module, a regression task module, and a causal reasoning module; Inputting the multimodal feature vector into the multi-task joint learning model, outputting a category label through the classification task module, outputting a continuous prediction value through the regression task module, and outputting a causal relationship path through the causal reasoning module; Performing logical correction on the category labels and continuous prediction values based on the causal relationship path to generate a causal correction reasoning result; Match and verify the causal correction reasoning result with the association strategy in the domain knowledge graph to generate a verified reasoning result; Integrate the category labels, continuous prediction values, and causal relationship paths in the verified inference results to generate inference result data.

6. The decision-making method based on multimodal data according to claim 1, characterized in that: Updating the parameters of the reasoning model based on the corrected feedback data to generate an updated reasoning model includes: De-duplication and label alignment are performed on the corrected feedback data and the historical reasoning result data to generate an incremental training data set; Partially unfreeze the parameters of the inference model using an elastic weight consolidation algorithm, and update the unfrozen parameters based on the incremental training data set to generate updated model parameters; Distributing the updated model parameters to multiple verification nodes, performing reasoning performance verification based on the local data set of each verification node, and generating verification result indicators; Determine the sensitivity and specificity of the model based on the validation result indicators, and select updated model parameters that meet the preset thresholds; Based on the filtered updated model parameters and frozen parameters, a new version of the inference model is built and the model registry is updated.

7. The decision-making method based on multimodal data according to claim 1, characterized in that: After updating the parameters of the inference model based on the corrected feedback data to generate an updated inference model, the method further includes: Performing symmetric encryption algorithm processing on the desensitized time series data, anonymized image data and encrypted annotation data in the standardized data to generate encrypted standardized data; Performing homomorphic encryption on the multimodal feature vector to generate an encrypted multimodal feature vector; Performing hash desensitization processing on the privacy attribute field in the inference result data to generate a hash desensitized inference result; Performing dynamic masking processing on the sensitive information in the corrected feedback data to generate dynamic masked feedback data; Performing key sharding encryption processing on the parameter file of the updated inference model to generate encrypted model parameters; Deploy access control policies on data access interfaces and record access requests to the encrypted standardized data, encrypted multimodal feature vectors, hash desensitized inference results, dynamic mask feedback data, and encryption model parameters; Generate an audit log of the operation type, timestamp, and user identity information of the access request; The audit log is hashed and distributedly stored using blockchain technology to generate tamper-proof audit trail records.

8. A decision-making device based on multimodal data, characterized in that: The decision-making device based on multimodal data includes: A data preprocessing module is used to collect original multimodal data and perform cleaning and annotation processing on the original multimodal data to obtain cleaned and annotated data; A data standardization module is used to perform format conversion and desensitization processing on the cleaned and annotated data to generate standardized data; A multimodal feature extraction module, configured to perform multimodal feature extraction and fusion processing on the standardized data to generate a multimodal feature vector; An inference model module, configured to perform inference processing based on the multimodal feature vector through an inference model to generate inference result data; A feedback collection module, configured to push the inference result data to a review terminal and receive correction feedback data returned by the review terminal; The model updating module is used to update the parameters of the inference model based on the corrected feedback data to generate an updated inference model.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a decision program based on multimodal data stored in the memory and capable of running on the processor. When the decision program based on multimodal data is executed by the processor, the steps of the decision method based on multimodal data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a decision program based on multimodal data, and when the decision program based on multimodal data is executed by the processor, the steps of the decision method based on multimodal data as described in any one of claims 1 to 7 are implemented.

Images