Cloud platform-based multi-modal health medical data management method and system

Abstract

This application discloses a cloud-based multimodal health and medical data governance method and system, relating to the field of medical data processing. Firstly, through standardized preprocessing and cross-modal feature co-coding, it unifies the mapping of structurally diverse image visual information and text semantic information into a high-dimensional vector space. Based on this, a self-supervised strategy based on contrastive learning is employed to automatically explore and establish the initial correspondence between image 3D regions and textual description fragments. Finally, to address the diversity and ambiguity of doctors' descriptions and ensure the accuracy and reliability of the final alignment results, a human-machine collaborative calibration mechanism is introduced to professionally confirm and correct the machine-generated initial alignment. In this way, massive, discrete raw medical data can be efficiently and accurately governed into a structured, highly correlated, high-quality dataset, providing a solid data foundation for subsequent applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical data processing, and more specifically, to a multi-modal health medical data management method and system based on a cloud platform. BACKGROUND

[0002] With the rapid development of medical informatization and artificial intelligence technology, a large amount of multi-modal health medical data has been generated in hospital information systems (HIS), picture archiving and communication systems (PACS), etc., among which the DICOM format medical images and the supporting radiology text reports are the most core. These data contain great clinical and scientific research value, however, they are usually stored in isolation in different systems, and there is a lack of effective association between data, forming data islands. In order to break down the barriers and fully tap the value of data, especially in training accurate medical AI models, it has become an urgent need in the industry to build a unified platform that can integrate, process and deeply analyze these heterogeneous data. Cloud platforms, with their powerful computing power, scalable storage resources and convenient collaboration features, provide an ideal infrastructure for centralized management of large-scale multi-modal medical data.

[0003] However, in the field of medical data management, although some solutions attempt to gather data in the cloud, most still remain at the level of data storage, backup and simple retrieval. The existing systems do not fully consider the deep semantic association between image data and text reports in their design, and lack a mechanism to automatically establish accurate mapping between the two. When doctors review images and write reports, they complete the alignment of image features and textual descriptions in their minds, but this valuable association information is not systematically recorded and utilized. This fragmentation at the data level means that even with vast amounts of data, it is not possible to directly form high-quality image-text pair annotation datasets that can be used for AI model training, greatly limiting the depth of application and development speed of medical artificial intelligence technology. The core technical bottleneck of the above challenges is concentrated on the problem of fine-grained semantic alignment between image data and text reports. Specifically, image data is a collection of visual information based on spatial coordinates, while text reports are linear descriptions based on natural language, with a huge semantic gap between the two. At the same time, doctors have different report writing styles, with descriptions often containing ambiguous relative positions and diverse expressions, and a report may involve multiple lesions, forming a complex one-to-many or many-to-many correspondence with multiple regions in the image.

[0004] Therefore, how to cross the modal difference and automatically and accurately match the text segments describing specific lesions in the report with the corresponding three-dimensional regions in the image is the key to realizing the deep management and value transformation of multi-modal medical data, and is also a difficult problem that needs to be solved by current technology. SUMMARY

[0005] To address the challenge of lacking fine-grained semantic alignment between image data and text reports as mentioned in the background art, this application provides a cloud-based multimodal health and medical data governance method, comprising: performing normalized preprocessing on the acquired original DICOM sequence and original text report to obtain a preprocessed image tensor and a segmented text fragment set; performing cross-modal feature co-encoding on the preprocessed image tensor and the segmented text fragment set to obtain an image region embedding vector set and a text fragment embedding vector set; performing self-supervised preliminary alignment on the image region embedding vector set and the text fragment embedding vector set based on contrastive learning to obtain an initial alignment mapping; and performing human-machine collaborative calibration and confidence quantization on the initial alignment mapping based on the preprocessed image tensor and the segmented text fragment set to obtain a calibrated alignment mapping.

[0006] According to another aspect of this application, a cloud-based multimodal health and medical data governance system is provided, comprising: a normalization preprocessing module for normalizing and preprocessing the acquired original DICOM sequence and original text report to obtain a preprocessed image tensor and a segmented text fragment set; a cross-modal embedding module for performing cross-modal feature co-encoding on the preprocessed image tensor and the segmented text fragment set to obtain an image region embedding vector set and a text fragment embedding vector set; a preliminary alignment module for performing self-supervised preliminary alignment of the image region embedding vector set and the text fragment embedding vector set based on contrastive learning to obtain an initial alignment mapping; and a calibration module for performing human-machine collaborative calibration and confidence quantization on the initial alignment mapping based on the preprocessed image tensor and the segmented text fragment set to obtain a calibrated alignment mapping.

[0007] Compared with existing technologies, this application provides a cloud-based multimodal health and medical data governance method and system, which aims to solve the problem of lack of fine-grained semantic alignment between image data and text reports as mentioned in the background technology. Specifically, firstly, through normalized preprocessing and cross-modal feature co-encoding, the structurally disparate visual information of images and semantic information of text are uniformly mapped into a high-dimensional vector space, thereby effectively bridging the semantic gap between the two. On this basis, a self-supervised strategy based on contrastive learning is adopted to automatically explore and establish the initial correspondence between three-dimensional regions of images and text description fragments, solving the problem of lack of embedded links in existing systems. Finally, to address the diversity and ambiguity of doctors' descriptions and ensure the accuracy and reliability of the final alignment results, a human-machine collaborative calibration mechanism is introduced to professionally confirm and correct the initial alignment generated by the machine. In this way, massive, discrete raw medical data can be efficiently and accurately governed into a structured, highly correlated, high-quality dataset, providing a solid data foundation for subsequent applications. Attached Figure Description

[0008] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings.

[0009] Figure 1 This is a flowchart of a cloud-based multimodal health and medical data governance method according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of data flow in a cloud-based multimodal health and medical data governance method according to an embodiment of this application.

[0011] Figure 3 This is a flowchart of step 1 in the cloud-based multimodal health and medical data governance method according to an embodiment of this application.

[0012] Figure 4 This is a flowchart of step 3 in the cloud-based multimodal health and medical data governance method according to an embodiment of this application.

[0013] Figure 5 This is a flowchart of step 4 in the cloud-based multimodal health and medical data governance method according to an embodiment of this application.

[0014] Figure 6 This is a block diagram of a cloud-based multimodal health and medical data governance system according to an embodiment of this application. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. It should be understood that the drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] To address the technical challenges mentioned above, this application proposes a multimodal health and medical data governance method based on a cloud platform. Figure 1 This is a flowchart of a cloud-based multimodal health and medical data governance method according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the data flow of a cloud-based multimodal health and medical data governance method according to an embodiment of this application. Figure 1 and Figure 2As shown, the cloud-based multimodal health and medical data governance method according to an embodiment of this application includes: Step 1, performing normalization preprocessing on the acquired original DICOM sequence and original text report to obtain a preprocessed image tensor and a segmented text fragment set; Step 2, performing cross-modal feature co-encoding on the preprocessed image tensor and the segmented text fragment set to obtain an image region embedding vector set and a text fragment embedding vector set; Step 3, performing self-supervised preliminary alignment based on contrastive learning on the image region embedding vector set and the text fragment embedding vector set to obtain an initial alignment mapping; Step 4, performing human-machine collaborative calibration and confidence quantization on the initial alignment mapping based on the preprocessed image tensor and the segmented text fragment set to obtain a calibrated alignment mapping.

[0017] In step 1, the acquired raw DICOM sequences and raw text reports undergo normalization preprocessing to obtain preprocessed image tensors and segmented text fragment sets. It should be understood that in the governance and application of multimodal health and medical data, raw data often exhibits high heterogeneity and non-standardization. Medical images originating from different medical institutions and different models of scanning equipment show significant differences in resolution, slice thickness, pixel spacing, and even imaging protocols; correspondingly, text reports also vary greatly in style due to differences in radiologists' personal writing habits, terminology preferences, and report templates. Furthermore, this raw data contains a large amount of patient privacy information that requires strict protection. This raw data state is chaotic and disordered, making it extremely difficult, if not impossible, to directly use it for downstream deep learning model training or precise clinical analysis. Therefore, before conducting any in-depth cross-modal correlation analysis, these raw data need to undergo a series of normalization preprocessing steps. The purpose is to transform these diverse and formatted image and text data into a unified, clean, and standardized data representation, thereby building a solid and consistent data foundation for subsequent feature extraction and semantic alignment. This effectively bridges the gap between modalities and ensures the accuracy and reliability of subsequent processing.

[0018] In a specific example of this application, Figure 3 This is a flowchart of step 1 in the cloud-based multimodal health and medical data governance method according to an embodiment of this application. Figure 3As shown, step 1 involves normalizing and preprocessing the acquired original DICOM sequence and original text report to obtain a preprocessed image tensor and a segmented set of text fragments. This includes: step 11, extracting and sorting metadata from the original DICOM sequence to obtain a sorted stack of slices; step 12, reconstructing 3D volume data based on the sorted stack of slices to obtain reconstructed 3D volume data and voxel spacing metadata; and step 13, performing 3D volume data intensity transformation and spatial normalization on the reconstructed 3D volume data and voxel spacing metadata to obtain the preprocessed image tensor.

[0019] In a specific example of this application, step 1, which involves normalizing and preprocessing the acquired original DICOM sequence and original text report to obtain the preprocessed image tensor and the segmented text fragment set, further includes step 14, which involves desensitizing the original text report and performing semantic segmentation on the original text report to obtain the segmented text fragment set.

[0020] In the specific example above, step 1 can be implemented as follows: First, the raw data to be processed needs to be obtained, namely the raw DICOM sequence and the raw text report. The raw DICOM sequence refers to a collection of two-dimensional image files (usually with the .dcm extension) generated by medical imaging equipment such as CT and MRI during a complete scan. Each file represents a scan layer and contains pixel data and rich metadata information. The raw text report is a diagnostic report written by a radiologist for this imaging examination; it is an unstructured free text document. In a cloud platform environment, this raw data can be retrieved through standardized interfaces, such as by pulling image sequences from the hospital's Picture Archiving and Communication System (PACS) via the DICOMweb protocol, or by retrieving the corresponding text reports from the Radiology Information System (RIS) or Electronic Medical Record System (EMR) via a secure data interface, and centrally stored in a cloud-based object storage service for subsequent processing.

[0021] After acquiring the raw data, the first image processing step in the standardized preprocessing workflow, step 11, is executed. In practice, the processing task deployed on the cloud platform concurrently reads all independent DICOM files from a raw DICOM sequence. For each file, the metadata area in its header is parsed, and the key DICOM tags necessary for subsequent processing are precisely extracted. These tags mainly include: the pixel data corresponding to tag (7fe0, 0010), which is the core of the image itself; the pixel pitch corresponding to tag (0028, 0030), which defines the physical size of a single pixel in the X and Y directions within the image plane, for example, [0.7 mm, 0.7 mm]; the slice thickness corresponding to tag (0018, 0050), representing the thickness of a single scan slice; and the crucial image position (patient) corresponding to tag (0020, 0032), which provides a three-dimensional coordinate (X, Y, Z), precisely defining the spatial position of the first pixel at the top left corner of the two-dimensional slice in the standard patient coordinate system. After extracting the metadata of all slices, all slices are globally sorted based on the Z-axis coordinates obtained from the image location (patient) labels, i.e., arranged in ascending order. This ensures that the slice sequence is correctly arranged in physical order from foot to head or head to foot. This set of slice data that has undergone precise spatial sorting constitutes the sorted slice stack.

[0022] Next, step 12, based on the sorted slice stack generated in the previous step, performs 3D volumetric data reconstruction, integrating the discrete 2D image sequence into continuous 3D spatial data. Specifically, following the order of the sorted slice stack, the pixel data of each 2D slice, i.e., a 2D array, is stacked in the new Z-axis direction, ultimately forming a large 3D array. This 3D array is the reconstructed 3D volumetric data, and each element is called a voxel, representing a point in 3D space. Simultaneously, the voxel spacing metadata of this 3D volumetric data needs to be accurately calculated and recorded. The spacing in the X and Y directions can be directly obtained from the previously extracted pixel spacing labels. The spacing in the Z direction, i.e., the interslice spacing, can be determined in two ways: one is to directly use the slice thickness label value; the other, more accurate and robust method, is to calculate the difference in the Z coordinates of the image position (patient) labels of two adjacent slices after sorting. This method accurately reflects the slice interval during the actual scan and can correctly handle even if scanning gaps exist. Ultimately, the voxel spacing metadata is organized into a list or tuple containing three values, such as [0.7 mm, 0.7 mm, 5.0 mm], which precisely describes the size of each voxel in the reconstructed 3D volume data in three physical dimensions.

[0023] After obtaining the reconstructed 3D volumetric data and voxel spacing metadata in step 13, it is necessary to perform 3D volumetric data intensity transformation and spatial standardization to ensure that all image data input into subsequent models has a consistent numerical range and spatial scale. The first step is intensity transformation, i.e., window width and window level adjustment. The voxel values ​​of the original CT images are expressed in Henle units (HU), which range widely, from -1024 to +3071. Different tissue structures are displayed most clearly within different HU value ranges. For example, lung windows are needed to observe lung structures, while bone windows are needed to observe bones. A window width (WW) and window level (WL) parameter are automatically selected or preset based on the examination site (obtainable from DICOM metadata). The determination process is as follows: First, specific tags in the DICOM metadata, such as examination description or examination site, are parsed. Then, the text information extracted from these tags is matched against a preset mapping rule base stored in a cloud configuration. This rule base defines the correspondence between keywords for different body parts and the optimal window width and window level values. For example, if the keywords "chest" or "lung" are identified from the examination description label, the preset lung window parameters are automatically applied, such as setting the window width to 1500 HU and the window level to -600 HU. Similarly, if the abdomen is identified, the abdominal window parameters (e.g., window width 400 HU, window level 50 HU) are applied. To ensure the robustness of the process, when the examination site cannot be clearly identified from the metadata, a set of general soft tissue default window width and window level parameters can be applied. After determining the parameters, each voxel value (V_in) in the 3D volume data is processed using the following linear transformation formula: when V_in is less than or equal to WL-WW / 2, the output value V_out is 0 (pure black); when V_in is greater than WL+WW / 2, the output value V_out is 255 (pure white); when V_in is between the two, its value is linearly mapped to the grayscale range of 0 to 255. This transformation maximizes the exposure of the tissue details of interest while normalizing voxel values ​​to a standard 8-bit integer range. Next, spatial normalization, also known as spatial resampling, is performed. Due to differences in parameters across different scanning devices, the voxel spacing metadata obtained in the previous step may be anisotropic, such as [0.7, 0.7, 5.0]. This inconsistency can severely impact the performance of models such as 3D convolutional neural networks. Therefore, a uniform target spacing is set, for example, an isotropic [1.0 mm, 1.0 mm, 1.0 mm]. Then, using a 3D interpolation algorithm, such as trilinear interpolation which balances computational efficiency and performance, the intensity-transformed 3D volume data is resampled based on the original voxel spacing and the target spacing, calculating the voxel values ​​at the new, uniform grid points. This process not only unifies the physical scale of all image data but also ensures that the image size is no longer directly linked to the number of original slices, but is determined by both its physical volume and the target spacing.After the two sub-steps of intensity transformation and spatial normalization, the output is a three-dimensional array with normalized dimensions, numerical range, and spatial scale, which is the preprocessed image tensor.

[0024] While processing the image data, step 14 also performs text anonymization and semantic segmentation on the original text report to protect patient privacy and extract the smallest semantic units for alignment. The first step is text anonymization. The original report inevitably contains personally identifiable information (PII) such as patient name, ID number, contact information, and specific examination date. The original text report is then input into a privacy information recognition and replacement module. The core of this module can be a deep learning-based Named Entity Recognition (NER) model. This NER model uses a Transformer architecture, such as BERT, and has been fine-tuned on a large-scale medical text corpus annotated with privacy information. The model's weights and bias parameters are learned during this fine-tuning process using a backpropagation algorithm, with the goal of minimizing the cross-entropy loss function of entity recognition. When a new text is input, the model can accurately identify predefined privacy entity categories. For example, the identified privacy information could be: Patient Zhang San was admitted to the hospital on October 26, 2023. The module will replace it with a uniform placeholder without specific information, such as [name], [date], and the processed text becomes: Patient [name] was admitted on [date]. In addition to the model method, the module can also be supplemented with a rule base based on regular expressions to match information with fixed formats such as ID card number, telephone number, etc., to improve the coverage and accuracy of desensitization. After desensitization is completed, the semantic segmentation stage is entered. A complete radiology report contains multiple chapters such as examination site, imaging findings, and diagnostic opinions. Among them, the imaging findings section is the core content describing the specific abnormalities found in the images. Therefore, firstly, coarse-grained chapter segmentation is performed based on these common chapter title keywords to accurately extract the text content of the imaging findings section. Then, fine-grained semantic segmentation is performed on this descriptive text. The segmentation rules take into account a variety of linguistic cues, including sentence-end punctuation marks, such as period ".", semicolon ";", line breaks, and paragraph-level numbers, such as "1.", "2.", "(1)", etc. These rules divide what was originally a continuous block of text into a list of multiple independent, grammatically and semantically relatively complete short sentences or phrases. For example, the original text "A nodular shadow measuring approximately 2.1 cm * 1.5 cm with rough edges was visible in the upper lobe of the right lung; no enlarged lymph nodes were seen in the bilateral hilum" will be divided into two independent text segments: "A nodular shadow measuring approximately 2.1 cm * 1.5 cm with rough edges was visible in the upper lobe of the right lung" and "No enlarged lymph nodes were seen in the bilateral hilum." The final generated set, consisting of a series of anonymized short texts describing specific imaging features, is the segmented text fragment set.

[0025] In step 2, cross-modal feature co-encoding is performed on the preprocessed image tensor and the segmented text fragment set to obtain the image region embedding vector set and the text fragment embedding vector set, respectively. Correspondingly, after the normalization preprocessing in the previous stage, although the original DICOM sequence and text report have been transformed into structurally unified image tensors and segmented text fragment sets, a fundamental gap still exists between the two in terms of data nature. Image tensors are three-dimensional spatial data based on pixel or voxel intensity values, and their information is encoded in visual patterns and spatial structures; while text fragment sets are one-dimensional linear sequences based on characters, and their information is carried on the grammatical and semantic rules of natural language. These two modalities of data cannot be directly compared or correlated meaningfully. Therefore, before achieving fine-grained semantic alignment, these two heterogeneous data sources need to be projected into a unified, machine-understandable representation space.

[0026] In a specific example of this application, step 2, performing cross-modal feature co-encoding on the preprocessed image tensor and the segmented text fragment set to obtain an image region embedding vector set and a text fragment embedding vector set, includes: step 21, inputting the preprocessed image tensor into a 3D convolutional neural network model pre-trained on medical images to obtain the image region embedding vector set; step 22, inputting the segmented text fragment set into a language model pre-trained on massive medical texts to obtain the text fragment embedding vector set.

[0027] In the specific example above, step 2 can be implemented as follows: First, step 21 encodes the features of the image data. The preprocessed image tensor generated in the previous stage is fed as a complete input into a 3D convolutional neural network model pre-trained on massive medical image data to extract its rich spatial hierarchical features and finally obtain the image region embedding vector set. In this embodiment, a 3D ResNet is selected as the image encoder. The specific architecture of this model is a direct extension of the classic 2D ResNet in 3D space. Its core advantage lies in building a very deep network by introducing residual learning units, which effectively solves the gradient vanishing and network degradation problems in deep network training. The 3D ResNet-50 architecture includes the following parts: an initial, large 3D convolutional layer for preliminary extraction of low-level features and reduction of spatial resolution; followed by four stages of residual block stacking, each stage containing multiple residual blocks, and downsampling between stages by 3D convolution with a stride of 2; finally, a global average pooling layer and a fully connected layer (but these two layers are not used in this application). Each residual block consists of two or three 3D convolutional layers, batch normalization layers, and a modified linear unit (ReLU) activation function. Crucially, it includes a shortcut connection that directly adds the block's input to its output, allowing the network to more easily learn identity mappings. The weights and biases of this 3D ResNet model are pre-trained on a large-scale public or private 3D medical image dataset (e.g., the DeepLesion dataset containing hundreds of thousands of CT scans). Pre-training tasks can include lesion classification, segmentation, or detection. Through this process, the model learns general prior knowledge about human anatomy, tissue texture, and common lesion patterns, giving it powerful visual feature representation capabilities. In the implementation, a pre-processed image tensor (e.g., representing a lung CT scan, with dimensions of 128x128x128) is input into this pre-trained 3D ResNet model. During forward propagation, the data passes sequentially through each convolutional layer and residual block of the network. Specifically, the goal of feature extraction is not a single global feature vector for the entire image, but rather local features that represent different spatial regions within the image. Therefore, this step does not take the output of the network's final stage, but instead extracts the feature map output from the last convolutional stage. This feature map is a four-dimensional tensor, whose dimensions can be represented as (number of channels, depth, height, width), for example (2048, 8, 8, 8). In this feature map, the vector (a vector of length 2048) at each spatial location (d, h, w) corresponds to a specific three-dimensional receptive field in the original input image, i.e., a deep semantic abstraction of a local region.To obtain the final image region embedding vector set, the spatial dimensions (depth, height, and width) of this four-dimensional feature map are flattened. Specifically, feature vectors from all locations on this 8x8x8 spatial grid are extracted one by one, forming a set containing 8*8*8=512 vectors. Each vector is a 2048-dimensional embedding vector, which condenses all the visual information of a specific three-dimensional sub-region in the original image. This set of 512 high-dimensional vectors is the final generated image region embedding vector set.

[0028] Simultaneously, step 22 performs feature encoding on the text data in parallel. Each text segment from the segmented text fragment set generated in the previous stage is input one by one into a language model pre-trained on a massive amount of medical text to parse its complex semantic information, thereby obtaining a text fragment embedding vector set. In this embodiment, a BERT model specifically optimized for the clinical medical field, namely ClinicalBERT, is selected as the text encoder. The BERT model is based on the Transformer architecture, and its core is a multi-head self-attention mechanism, which can capture long-distance dependencies in text sequences and understand the meaning of each word bidirectionally according to the context. A standard BERT-base model contains 12 Transformer encoder layers, each with a hidden state dimension of 768 and 12 attention heads. Similar to the image encoder, the parameters of the ClinicalBERT model are also obtained through large-scale pre-training. It first inherits the basic language knowledge learned by the general BERT on general corpora such as Wikipedia, and then performs secondary pre-training on a huge professional corpus (such as the MIMIC-III database) containing millions of clinical records and medical literature. The pre-training task mainly includes masked language modeling, which predicts randomly masked words in a sentence and the next sentence. Through this domain-adaptive pre-training, ClinicalBERT's understanding of medical terminology, abbreviations, syntactic structures, and clinical logic far surpasses that of general language models. In the specific implementation process, each element in the segmented text fragment set is traversed. Taking the fragment "A nodular shadow of approximately 2.1 cm * 1.5 cm in size with rough edges is visible in the upper lobe of the right lung" as an example, firstly, this text fragment is fed into ClinicalBERT's dedicated word segmenter for processing. The word segmenter divides the text into a series of tokens based on a large vocabulary, adding a special classification marker [CLS] at the beginning of the sequence and a separator marker [SEP] at the end. Then, these tokens are converted into corresponding numerical indices and constructed into the input format required by the model. Next, this processed input sequence is fed into the ClinicalBERT model. The data passes through 12 Transformer encoder layers within the model. At each layer, a self-attention mechanism recalculates the representation of each word based on the global context. Finally, the model outputs the hidden state vector of all words in the last layer. Following standard BERT usage, the output vector corresponding to the input [CLS] tag is extracted. This 768-dimensional vector is considered the aggregate semantic representation of the entire text segment because it is used during training to capture the overall information of the sequence for downstream tasks such as classification. This vector is the embedding vector of the text segment.This process is repeated for all segments in the text fragment set. For example, the same operation is performed on "no enlarged lymph nodes were seen in the bilateral hilum of the lungs," resulting in its corresponding 768-dimensional embedding vector. Finally, the embedding vectors generated from all text fragments are collected together to form a set, which is the text fragment embedding vector set.

[0029] Through the parallel encoding process of images and text described above, the originally heterogeneous and incomparable sets of image tensors and text fragments were successfully transformed into two sets of embedding vectors: an image region embedding vector set, where each vector represents a three-dimensional region in the image; and a text fragment embedding vector set, where each vector represents a description in the report. Although the vectors in these two sets may have different dimensions—for example, 2048 dimensions for images and 768 dimensions for text—they both exist in a unified, high-dimensional semantic feature space. In this space, semantically related image regions and text descriptions will have their corresponding embedding vectors more spatially close.

[0030] In step 3, a self-supervised preliminary alignment based on contrastive learning is performed on the image region embedding vector set and the text fragment embedding vector set to obtain an initial alignment map. It is understood that after cross-modal feature co-encoding, the original, heterogeneous image and text data have been successfully transformed into two sets of embedding vectors in a unified high-dimensional semantic space. However, this only establishes the mathematical foundation for subsequent comparison and association; the specific one-to-one correspondence between the two vector sets remains unknown. How to accurately find the most visually relevant region for each text fragment describing a specific lesion from vectors representing hundreds of different three-dimensional image regions is the core challenge of achieving fine-grained semantic alignment and the key to solving the semantic gap and complex correspondence problems mentioned in the background art. Therefore, this application uses a self-supervised preliminary alignment mechanism based on contrastive learning to automatically and on a large scale calculate and evaluate all possible text-image pairings, quantify their semantic association strength, and select the matching relationship with the highest confidence, thereby generating a preliminary, machine-readable alignment map table to provide high-quality initial input for subsequent human-machine collaborative calibration.

[0031] In a specific example of this application, Figure 4 This is a flowchart of step 3 in the cloud-based multimodal health and medical data governance method according to an embodiment of this application. Figure 4As shown, step 3, performing self-supervised preliminary alignment based on contrastive learning on the image region embedding vector set and the text fragment embedding vector set to obtain the initial alignment mapping, includes: step 31, calculating the cosine similarity between any pair of image region embedding vectors and text fragment embedding vectors in the image region embedding vector set and the text fragment embedding vector set to obtain a cross-modal cosine similarity matrix; step 32, performing Softmax activation on the cross-modal cosine similarity matrix to obtain a region attention vector set; step 33, based on the region attention vector set, performing optimal matching localization and initial mapping generation on the image region embedding vector set and the text fragment embedding vector set to obtain the initial alignment mapping.

[0032] In the specific example above, step 3 can be implemented as follows: Step 31 comprehensively quantifies the semantic correlation between each text description and each image region. In the preceding steps, an image region embedding vector set has been obtained, which contains multiple high-dimensional vectors representing different three-dimensional spatial regions in the image; and a text fragment embedding vector set, which contains multiple high-dimensional vectors representing different descriptive statements in the report. For ease of explanation, in this application, since the feature vector dimension of 2048 output by the image encoder (such as 3DResNet) and the feature vector dimension of 768 output by the text encoder (such as ClinicalBERT) are usually inconsistent, similarity calculation cannot be performed directly. Therefore, a linear projection layer is needed for dimension alignment. This linear projection layer is essentially a simple fully connected neural network layer, which contains a learnable weight matrix and a bias vector. For each input high-dimensional vector, this layer performs a matrix multiplication and adds the bias, thereby linearly transforming or projecting it into a target space of a preset dimension. The weights and bias parameters of this projection layer are learned through backpropagation during the training of the entire cross-modal alignment model, with the goal of maximizing the similarity of matching text-image pairs and minimizing the similarity of mismatched text-image pairs. By applying their respective linear projection layers to the outputs of the two modalities, the dimensions of both the image region embedding vector and the text fragment embedding vector are unified to 512 dimensions. The image region embedding vector set contains 512 vectors, each with 512 dimensions, denoted as {v_1,v_2,...,v_512}; the text fragment embedding vector set contains two vectors, each with 512 dimensions, corresponding to "a nodular shadow of approximately 2.1 cm * 1.5 cm in size with rough edges is visible in the upper lobe of the right lung" and "no enlarged lymph nodes are seen in the bilateral hilum," denoted as {t_1,t_2}. Cosine similarity is an index that measures the degree of similarity between two vectors in direction, with a value range of [-1,1]. The closer the value is to 1, the more consistent the directions of the two vectors are, that is, the more similar their semantics are. The calculation process involves iterating through each text vector in the text fragment embedding vector set and pairing it with each image vector in the image region embedding vector set, calculating the cosine similarity between them. Specifically, for the first text vector t_1, the cosine similarity between it and v_1, v_2, ..., v_512 is calculated, resulting in a sequence of 512 similarity values ​​(S_11, S_12, ..., S_1, 512). Similarly, the same calculation is performed on the second text vector t_2, resulting in another sequence of 512 similarity values ​​(S_21, S_22, ..., S_2, 512). These two sequences are stacked together as rows, forming a 2x512 matrix. This matrix is ​​the cross-modal cosine similarity matrix, denoted as S.Each element S_ij in the matrix precisely represents the semantic correlation strength between the i-th text fragment and the j-th image region.

[0033] Step 32 involves applying Softmax activation to the cross-modal cosine similarity matrix generated in the previous step to obtain a set of region attention vectors. While the original cosine similarity values ​​can represent the association strength, their numerical distribution is not intuitive and difficult to directly interpret as a matching probability. The purpose of Softmax activation is to transform these original similarity scores into a normalized, probability-like attention weight distribution. This allows the association strength between each text segment and all image regions to be clearly presented in the form of a relative weight with a sum of 1. In a specific example of this application, step 32, applying Softmax activation to the cross-modal cosine similarity matrix to obtain a set of region attention vectors, includes: applying Softmax activation to the cross-modal cosine similarity matrix using the following formula: ;in, For temperature coefficient, and The first element in the cross-modal cosine similarity matrix In the middle of the line The and the first One value, It is the number of columns in the cross-modal cosine similarity matrix. It is the first in the set of region attention vectors The th region attention vector The attention value for each region. Sij is the value in the i-th row and j-th column of the cross-modal cosine similarity matrix, which is the similarity between the i-th text segment and the j-th image region. It is a key hyperparameter known as the temperature coefficient. Its role is to adjust the sharpness of the attention distribution. When When the value of τ is small (e.g., close to 0), the Softmax function greatly amplifies the differences between similarity scores, making the probability of the highest score close to 1, while the probabilities of other scores are close to 0, forming a very sharp distribution. This helps in making clear decisions when differences are significant. When the value of τ is large, the function softens the probability distribution, making the probability differences between different scores smaller and the distribution smoother. This helps retain more possibilities when multiple regions are related to the text. The value of needs to be set empirically or optimized experimentally based on the specific task and data characteristics. A commonly used reference value in the field of contrastive learning is 0.07. M is the number of columns in the cross-modal cosine similarity matrix, which is 512 in this example, representing the total number of image regions. The numerator of the formula... All similarity values ​​(including negative values) are mapped to the positive domain via an exponential function, modulated by a temperature coefficient τ. The denominator... This involves summing the exponential similarity scores between the i-th text segment and all M image regions, serving a normalization function. In practice, this Softmax function is applied to each row (corresponding to a text segment) of the cross-modal cosine similarity matrix S. For example, for the first row (S_11,S_12,...,S_1,512), a new vector A_1=(A_1(1),A_1(2),...,A_1(512)) is obtained, where the sum of all elements is 1. This vector A_1 is the attention distribution of the first text segment across all image regions. Similarly, A_2 is obtained by calculating the second row. These two attention vectors A_1 and A_2 together form the set of region attention vectors.

[0034] Specifically, after obtaining the cross-modal cosine similarity matrix, the next step is to transform these raw similarity scores into normalized attention weights that can guide matching and localization. However, in real-world medical scenarios, the specificity of text descriptions varies greatly. For example, some descriptions are anatomically precise and specific, such as "a nodule measuring approximately 2.1 cm * 1.5 cm with rough edges is visible in the upper lobe of the right lung," and their corresponding similarity score distribution should exhibit a sharp, unimodal shape (low entropy, high variance); while other descriptions are ambiguous and generalized, such as "no enlarged lymph nodes were seen in the bilateral hilum," and their corresponding similarity score distribution may exhibit a flat, multimodal shape (high entropy, low variance). If a fixed temperature coefficient is used for globally uniform Softmax activation, for highly specific descriptions, the localization accuracy may be weakened due to excessively high temperature; for ambiguous descriptions, the distribution may be forcibly sharpened due to excessively low temperature, resulting in a potentially incorrect and overconfident prediction. Therefore, to enable the attention mechanism to intelligently adapt to the inherent properties of different text descriptions, this application should perform dynamic temperature scaling based on the statistical characteristics of the similarity distribution. Specifically, it should perform adaptive temperature adjustment on the cross-modal cosine similarity matrix based on specificity variance to obtain a more accurate and interpretable set of region attention vectors. The aim of this method is to dynamically calculate a unique temperature value for each text segment, allowing the Softmax function to use a lower temperature for highly specific descriptions to achieve precise localization, while using a higher temperature for ambiguous descriptions to reflect their inherent uncertainty, thereby generating more refined and robust region attention vectors.

[0035] In a preferred example of this application, step 32, performing Softmax activation on the cross-modal cosine similarity matrix to obtain a set of region attention vectors, includes:

[0036] The cross-modal cosine similarity matrix is ​​structurally represented and spatially normalized to obtain the structural representation matrix, i.e.: ;in, It is a cross-modal cosine similarity matrix. yes The transpose of the matrix, To calculate the F-norm of a matrix, It is a structural representation matrix. First, it captures and quantifies the matching complexity and inherent correlation of the current text-image data to the whole, thus providing a global benchmark for subsequent evaluation of the specificity of individual text descriptions. Specifically, the formula transforms the original NxM-dimensional cross-modal cosine similarity matrix... Rather than transpose Matrix multiplication yields an NxN square matrix that reflects the relationships between similarity distribution patterns of different text fragments. Subsequently, the F-norm of matrix S is used to normalize this matrix, generating a dimensionally regular and numerically stable structural representation matrix. It encapsulates the global statistical properties of the entire similarity matrix.

[0037] The standard deviation of the structural representation matrix is ​​calculated to obtain the structural standard deviation. Then, the global properties obtained in the previous step are refined into a single scalar value that can be used for normalization. Specifically, the structural representation matrix is ​​calculated. The standard deviation of all elements in the dataset is used to produce a global structural standard deviation. This value will serve as a benchmark for subsequent measurements of local specificity, ensuring that the judgment of specificity is not isolated but relative to the overall distribution of the current data pair.

[0038] Calculate the variance and standard deviation of each similarity vector in the cross-modal cosine similarity matrix. Then, directly quantify the core specificity of each independent text description. Specifically, for the cross-modal cosine similarity matrix... The i-th row represents the similarity vector of the i-th text segment. Calculate their variances respectively. and standard deviation This approach utilizes variance as a statistical metric to capture the distribution pattern of similarity scores: high variance typically indicates a wide score distribution, suggesting a clear peak far from the mean, corresponding to highly specific matches; while low variance indicates that many scores cluster together, suggesting ambiguity and no clear best match. This generates a metric for each text segment that quantifies the degree of ambiguity in its matching.

[0039] After dividing the variance of each similarity vector by the structural standard deviation to obtain the specificity statistical representation, multiplying it by the standard deviation of the corresponding similarity vector to obtain the specificity variance representation; then applying an exponential decay function to dynamically temperature scale each specificity variance representation to obtain the temperature-scaled value, i.e.: ;in, This is a hyperparameter representing the baseline or maximum temperature. When the variance is very low (i.e., highly fuzzy), the temperature will approach this value. It is a positive scaling hyperparameter used to control the sensitivity of temperature to changes in the specificity variance representation; the higher the value, the better. This means that as the difference in similarity score distribution increases, the temperature will drop faster. It is the variance of each similarity vector. It is the structural standard deviation. It is the standard deviation of each similarity vector. These are the various temperature scaling values. Then, a more robust and sensitive specificity metric is constructed. Specifically, through... This calculation first considers the local variance. Using global structural standard deviation Normalization is performed to eliminate the impact of differences in the overall distribution between different data points; then it is multiplied by the standard deviation of the vector itself. This further amplifies the specificity signal, making the final representation more sensitive to changes in distribution morphology. This generates a refined, multi-calibrated specificity variance representation for each text segment. Next, the specificity metric calculated in the previous step is mapped to a temperature value ultimately used for Softmax activation through a non-linear, well-controlled function. Specifically, this is achieved by... .in, It is a hyperparameter representing the base or maximum temperature. When the specific variance of a description is very low (i.e., highly fuzzy), the exponential term approaches 1, making the temperature... It should approach this base value, for example, it can be set to 1.0. These are positive scaling hyperparameters used to control the sensitivity of the specificity variance representation to changes in temperature; for example, they can be set to 0.5. The values ​​of these two hyperparameters are determined through experimental tuning on the validation set. This way, for descriptions with large specificity variance representations (high specificity), the exponential term becomes very small, leading to... A lower value makes the Softmax function harder; conversely, a higher value indicates a smaller specific variance (high fuzziness). Approaching This makes the Softmax function softer.

[0040] Based on various temperature scaling values, Softmax activation is applied to each similarity vector in the cross-modal cosine similarity matrix to obtain a set of region attention vectors, i.e.: ;in, and The first element in the cross-modal cosine similarity matrix In the middle of the line The and the first The value, i.e., the th value. A similarity vector, It is the number of columns in the cross-modal cosine similarity matrix. It is the first in the set of region attention vectors The th region attention vector Attention values ​​for each region. Finally, the final step of applying the aforementioned dynamic temperature adjustment mechanism. Specifically, for each row of the similarity matrix... They all use their own proprietary, dynamically calculated temperature values. Softmax calculation is then performed. For highly specific descriptions, the generated region attention vector will exhibit a very sharp peak at the optimal matching position, resulting in more accurate localization of explicit matches and higher confidence scores. For ambiguous descriptions, a flatter region attention vector will be generated, with its probability distributed across multiple possible regions. This accurately reflects the inherent uncertainty and prevents the model from making overconfident and erroneous judgments when processing ambiguous inputs.

[0041] Step 33, based on the region attention vector set generated in the previous step, performs optimal matching localization and initial mapping on the image region embedding vector set and the text fragment embedding vector set to interpret the attention distribution and form a structured alignment result, that is, for each text description, finds its most likely corresponding physical region in the three-dimensional image space. It iterates through each region attention vector in the region attention vector set. For the first region attention vector A_1 (which corresponds to the text fragment "a nodular shadow of approximately 2.1 cm * 1.5 cm in size with rough edges is visible in the upper lobe of the right lung"), it searches for the maximum value among its 512 attention values. The search reveals that A_1(150) is the maximum value, with a value of 0.92. This maximum value of 0.92 is directly used as the confidence score for this alignment, quantifying the model's confidence in the match between the first text fragment and the 150th image region. Next, the specific image region needs to be located based on the index (j=150) of this maximum value. This index 150 directly corresponds to the 150th vector in the image region embedding vector set. More importantly, during the feature extraction stage, there is a one-to-one correspondence between the index of each image region vector and its spatial coordinates in the 3D feature map. For example, if the feature map has dimensions of 8x8x8 (depth x height x width), then the one-dimensional index 150 can be converted back to 3D coordinates using the formulas d=index / (H*W), h=(index%(H*W)) / W, w=index%W, where index is the index, / is integer division, and % is the remainder. In this example, 150 = 2*(8*8) + 2*8 + 6. That is, 2*(8*8) represents that before the 3rd layer (d=2), two complete layers (the 1st and 2nd layers) have been traversed. These two layers have a total of 2*64 = 128 squares. 2*8 represents that within the 3rd layer, before the 3rd row (h=2), two complete rows (the 1st and 2nd rows) have been traversed. These two rows have a total of 2*8 = 16 squares. The number 6 represents the 3rd layer, 3rd row, and it's located at the 7th position (w=6), so there are 6 squares before it. Adding these together: 128 (previous layer number) + 16 (previous row number) + 6 (previous column number) = 150. Therefore, its corresponding 3D feature map coordinates are (2,2,6). Using these feature map coordinates, and combining them with the network structure parameters of the 3D ResNet model, the 3D spatial range corresponding to this region in the original, high-resolution preprocessed image tensor is calculated in reverse. This inverse calculation mainly depends on the network's total downsampling factor (i.e., total stride). If the total stride from the model's input to this feature map is 16 in each dimension (i.e., 128 / 8=16), then one square on the feature map corresponds to a 16x16x16 voxel-sized region in the original image.The voxel coordinates of the center point of the region represented by the feature map coordinates (2,2,6) in the original image tensor can be approximately calculated as ((2+0.5)*16,(2+0.5)*16,(6+0.5)*16), i.e. (40,40,104). Therefore, a 16x16x16 voxel bounding box centered at this point can be defined, with minimum and maximum coordinates of (32,32,96) and (48,48,112). This range (x_min=32,y_min=32,z_min=96,x_max=48,y_max=48,z_max=112) precisely defines the location of the lesion described in the text within the standardized image space. Finally, the original text fragments ("A nodular shadow measuring approximately 2.1 cm * 1.5 cm with rough edges is visible in the upper lobe of the right lung"), their spatial coordinates (3D bounding box), and the calculated confidence score of 0.92 are encapsulated into a structured alignment record. This process is repeated for all vectors in the region attention vector set. For example, for the second region attention vector A_2 (corresponding to the text "No enlarged lymph nodes seen in the bilateral hilum"), its maximum attention value and its index are found, such as the maximum value at index 400, which is 0.70. Its corresponding 3D bounding box is calculated as (60, 60, 40, 76, 76, 56), and a second alignment record is generated. The alignment records generated from all text fragments are then aggregated to form a list or dictionary structure. This final set is the initial alignment map. It provides a specific spatial location on a 3D image for each meaningful text description, along with a confidence assessment. For example, the initial alignment mapping can be represented in the following structured data format: [{"text fragment":"A nodular shadow of about 2.1 cm * 1.5 cm in size is visible in the upper lobe of the right lung, with rough edges","confidence score":0.92,"spatial coordinates":{"coordinate system":"voxel coordinates","bounding box":[32,32,96,48,48,112]}},{"text fragment":"No enlarged lymph nodes were seen in the hilum of both lungs","confidence score":0.70,"spatial coordinates":{"coordinate system":"voxel coordinates","bounding box":[60,60,40,76,76,56]}}].

[0042] In step 4, based on the preprocessed image tensor and the segmented text fragment set, the initial alignment mapping is calibrated and quantified using a human-machine collaborative approach to obtain the calibrated alignment mapping. It should be understood that although the preceding steps generate an initial alignment mapping through a self-supervised learning mechanism, establishing preliminary semantic links for massive amounts of unlabeled text and image data, the product of this automated process is essentially a result based on model probabilistic inference. Given the rigor of the medical field and the complexity and variability of physician reporting language, the model's understanding is inevitably biased, and the confidence score it provides may not fully reflect the gold standard of clinical judgment. For example, the model might incorrectly locate a vaguely descriptive text sentence onto nearby normal tissue, or the given location bounding box might be inaccurate. Directly using this potentially erroneous alignment data to train downstream medical AI models will introduce noise, affecting the model's performance and reliability. Therefore, this application introduces the knowledge and judgment of domain experts to construct a human-machine collaborative closed-loop calibration process, which conducts targeted review, correction and confirmation of the preliminary results generated by the machine, thereby transforming the probabilistic association of the initial alignment mapping into a highly deterministic factual association endorsed by experts, and finally producing a high-quality calibrated alignment mapping that can be directly used for high-precision model training and clinical research.

[0043] In a specific example of this application, Figure 5 This is a flowchart of step 4 in the cloud-based multimodal health and medical data governance method according to an embodiment of this application. Figure 5 As shown, step 4, based on the preprocessed image tensor and the segmented text fragment set, performs human-computer collaborative calibration and confidence quantification on the initial alignment mapping to obtain the calibrated alignment mapping, including: step 41, according to the similarity score in the initial alignment mapping, push the alignment task to be corrected to the calibration interface, the alignment task to be corrected includes text fragments and corresponding image regions; step 42, receive the interpersonal interaction results input by the doctor object; step 43, perform confidence score correction based on the interpersonal interaction results to obtain the corrected confidence score.

[0044] In the specific example above, step 4 can be implemented as follows: Step 41 will push the alignment tasks that need correction to a specially designed calibration interface based on the similarity scores in the initial alignment mapping to achieve intelligent screening and focused review, rather than having experts conduct a comprehensive review of all results, thereby maximizing the use of valuable expert time. In specific implementation, one or more confidence thresholds are set. These confidence thresholds, as well as the subsequent manual intervention state mapping rules, are determined through consultation between algorithm engineers and domain experts after trial annotation and analysis of a small batch of validation datasets, aiming to balance the workload of review and the final data quality. For example, a low confidence threshold T_low, such as 0.6, and a high confidence threshold T_high, such as 0.9, can be set. All alignment records with confidence scores below T_low are considered results with extremely high model uncertainty and require manual review; all records with scores between T_low and T_high are considered results with moderate confidence and also need to be pushed for review; while records with scores above T_high are considered results with high model confidence and can be temporarily accepted or sampled for review with lower priority. The initial alignment map generated in this embodiment contains two records: the first is about "nodular shadows" with a confidence score of 0.92; the second is about "no enlarged lymph nodes seen in the bilateral hilum" with a confidence score of 0.70. Based on the aforementioned thresholds: the score of the first record (0.92) is higher than 0.9, therefore it is determined to be automatically received; the score of the second record (0.70) is between 0.6 and 0.9, therefore it is determined to be an alignment task requiring correction. At this point, all data related to this task is retrieved from cloud storage: the original text fragment "no enlarged lymph nodes seen in the bilateral hilum," its corresponding 3D bounding box coordinates [60,60,40,76,76,56] in the initial alignment map, and the complete preprocessed image tensor for this case. This data is packaged into a task unit and pushed in real-time to a web calibration interface designed for radiologists and other professionals via the cloud platform's message queue or API interface. The interface employs a multi-view layout. The left side clearly displays text snippets, while the right side features an interactive 3D image viewer. This viewer loads the complete image tensor and overlays the model-suggested 3D bounding boxes onto the corresponding locations on the image with semi-transparent highlights. Doctors can use the mouse to pan, zoom, rotate, and scroll through slices to examine the accuracy of the model's localization from any angle.

[0045] Next, step 42 receives the interpersonal interaction results input by the doctor, the subject of the operation, on the calibration interface. After reviewing the correspondence between the text and the image, the doctor will make a judgment based on their professional knowledge and provide feedback on their conclusion through the interactive controls provided on the interface. These controls offer at least three explicit operation options: First, "Confirm," indicating that the doctor fully agrees with the alignment relationship generated by the model, that is, the text description and the highlighted image area are perfectly matched; second, "Reject," indicating that the doctor believes this is a completely wrong match, and the content of the text description is unrelated to the highlighted area; third, "Adjust," indicating that the doctor believes the model's positioning direction is basically correct, but the bounding box is not precise enough, possibly too large, too small, or offset. If the doctor selects "Adjust," the interface will activate the bounding box editing function, allowing the doctor to manually correct its size and position by dragging the vertices or borders of the bounding box until they can accurately select the anatomical structure or lesion area described in the text. In the example of this application, the doctor sees the text "No enlarged lymph nodes seen in the bilateral hilar regions" and the bounding box of the model's positioning in the bilateral hilar regions. After carefully reviewing the images, the doctor determined that the model's localization of the region was correct, but the bounding box it provided was slightly too wide. Therefore, the doctor selected the "Adjust" option and manually fine-tuned the bounding box coordinates from [60,60,40,76,76,56] to a more compact [62,62,42,74,74,54]. After completing the adjustment, the doctor clicked the "Submit" button. At this point, the front-end interface encapsulates the doctor's interaction into a structured data packet, such as {"Interaction Type":"Adjusted","New Coordinates":[62,62,42,74,74,54]}, and sends it back to the cloud platform's backend.

[0046] Step 43 performs confidence score correction based on the received interpersonal interaction results to obtain a corrected confidence score, and updates the alignment mapping to quantify the expert's tacit knowledge and integrate it into the final data record, making it more accurate not only in spatial positioning but also more reliable in confidence assessment. In a specific example of this application, step 43, performing confidence score correction based on the interpersonal interaction results to obtain a corrected confidence score, includes: performing confidence score correction based on the interpersonal interaction results using the following formula, where the formula is: ;in, The original similarity score for the model. For variables related to the state of human intervention, As a weighting factor, This is the corrected confidence score. Specifically, It is the final output, corrected confidence score. This is the original similarity score given by the model, which is 0.70 in this application. This is a preset weighting factor, ranging from 0 to 1, representing the degree of confidence in the model's original score. This value can be set according to the level of conservatism required for the application scenario. For example, in a scenario that heavily relies on expert opinions... It can be set to a small value, such as 0.2, which means that the final score will be mainly determined by human intervention (accounting for 80% of the weight). This is a variable representing the state of human intervention. It maps the doctor's qualitative actions to quantitative values. This mapping rule is predefined and is a set of heuristic assignment standards jointly negotiated and set by domain experts and algorithm engineers based on the clinical judgment meaning implied by different human actions. For example, if the doctor's interaction result is "confirm," then... A value of 1.0 is assigned, representing complete correctness; if the result is "rejection", then... A value of 0.0 represents a complete error; if the result is "adjusted," which indicates that it is basically correct but has been slightly tweaked, a higher value between 0 and 1 can be assigned, such as 0.95, representing a high degree of reliability but not perfect. Substitute the values ​​from the example in this application into the formula for calculation: =0.70, The value is set to 0.2 because the doctor's operation is "adjustment". It is assigned a value of 0.95. Therefore, the corrected confidence score is: =0.2*0.70+(1-0.2)*0.95=0.90. After calculating the corrected confidence score, the corresponding record in the initial alignment mapping is updated. The confidence score in the original record is updated from 0.70 to 0.90, and the spatial coordinates are updated from [60,60,40,76,76,56] to [62,62,42,74,74,54] adjusted by the doctor. Through this series of human-machine collaborative processes, all alignment records pushed for review have been confirmed or corrected by experts. Finally, this collection, which includes all high-confidence automatically aligned records and all manually calibrated records, constitutes the final, high-quality calibrated alignment mapping. For example, the final calibrated alignment mapping can be represented in the following structured data format: [{"Text fragment":"A nodule measuring approximately 2.1 cm * 1.5 cm with rough edges is visible in the upper lobe of the right lung","Confidence score":0.92,"Spatial coordinates":{"Coordinate system":"Voxel coordinates","Bounding box":[32,32,96,48,48,112]},"Calibration status":"Automatic acceptance"},{"Text fragment":"No enlarged lymph nodes seen in the bilateral hilum","Confidence score":0.90,"Spatial coordinates":{"Coordinate system":"Voxel coordinates","Bounding box":[62,62,42,74,74,54]},"Calibration status":"Manual calibration"}]. This mapping data, with each record combining the efficiency of machine processing with the precision of human intelligence, provides a solid and reliable foundation for subsequent in-depth applications of medical data.

[0047] In summary, the cloud-based multimodal health and medical data governance method based on the embodiments of this application is explained, aiming to solve the problem of lack of fine-grained semantic alignment between image data and text reports as mentioned in the background art. Specifically, firstly, through normalized preprocessing and cross-modal feature co-encoding, the structurally disparate visual information of images and semantic information of text are uniformly mapped into a high-dimensional vector space, thereby effectively bridging the semantic gap between the two. On this basis, a self-supervised strategy based on contrastive learning is adopted to automatically explore and establish the initial correspondence between the three-dimensional regions of images and text description fragments, solving the problem of lack of embedded links in existing systems. Finally, to address the diversity and ambiguity of doctors' descriptions and ensure the accuracy and reliability of the final alignment results, a human-machine collaborative calibration mechanism is introduced to professionally confirm and correct the initial alignment generated by the machine. In this way, massive, discrete raw medical data can be efficiently and accurately governed into a structured, highly correlated, high-quality dataset, providing a solid data foundation for subsequent applications.

[0048] Figure 6 This is a block diagram of a cloud-based multimodal health and medical data governance system according to an embodiment of this application. Figure 6As shown, the cloud-based multimodal health and medical data governance system 100 according to an embodiment of this application includes: a normalization preprocessing module 110, used to perform normalization preprocessing on the acquired original DICOM sequence and original text report to obtain a preprocessed image tensor and a segmented text fragment set; a cross-modal embedding module 120, used to perform cross-modal feature co-encoding on the preprocessed image tensor and the segmented text fragment set to obtain an image region embedding vector set and a text fragment embedding vector set; a preliminary alignment module 130, used to perform self-supervised preliminary alignment based on contrastive learning on the image region embedding vector set and the text fragment embedding vector set to obtain an initial alignment mapping; and a calibration module 140, used to perform human-machine collaborative calibration and confidence quantization on the initial alignment mapping based on the preprocessed image tensor and the segmented text fragment set to obtain a calibrated alignment mapping.

[0049] Here, those skilled in the art will understand that the specific operations of each step in the aforementioned cloud-based multimodal health and medical data governance system have been referenced above. Figures 1 to 5 The description of the cloud-based multimodal health and medical data governance method is detailed here, and therefore, its repeated description will be omitted.

Claims

1. A multimodal health and medical data governance method based on a cloud platform, characterized in that, include: The acquired original DICOM sequence and original text report are normalized and preprocessed to obtain the preprocessed image tensor and the segmented text fragment set; Cross-modal feature co-encoding is performed on the preprocessed image tensor and the segmented text fragment set to obtain the image region embedding vector set and the text fragment embedding vector set; A self-supervised preliminary alignment based on contrastive learning is performed on the image region embedding vector set and the text fragment embedding vector set to obtain an initial alignment mapping; Based on the preprocessed image tensor and the segmented text fragment set, the initial alignment mapping is calibrated and quantized using human-machine collaborative calibration to obtain the calibrated alignment mapping. The process includes performing self-supervised preliminary alignment based on contrastive learning on the image region embedding vector set and the text segment embedding vector set to obtain an initial alignment mapping, including: Calculate the cosine similarity between any pair of image region embedding vectors and text fragment embedding vectors in the image region embedding vector set and the text fragment embedding vector set to obtain the cross-modal cosine similarity matrix; The method involves applying Softmax activation to the cross-modal cosine similarity matrix to obtain a set of region attention vectors. This includes: calculating the structural representation and spatial normalizing the cross-modal cosine similarity matrix to obtain a structural representation matrix, and calculating the standard deviation of the structural representation matrix to obtain the structural standard deviation; calculating the variance and standard deviation of each similarity vector in the cross-modal cosine similarity matrix; dividing the variance of each similarity vector by the structural standard deviation to obtain a specific statistical representation, and then multiplying it by the standard deviation of the corresponding similarity vector to obtain a specific variance representation; applying an exponential decay function to dynamically scale the specific variance representation to obtain a temperature scaling value; and applying Softmax activation to each similarity vector in the cross-modal cosine similarity matrix based on the temperature scaling value to obtain a set of region attention vectors. Based on the set of region attention vectors, the image region embedding vector set and the text fragment embedding vector set are used to perform optimal matching localization and initial mapping to generate the initial alignment mapping.

2. The multimodal health and medical data governance method based on a cloud platform according to claim 1, characterized in that, The acquired raw DICOM sequence and raw text report are subjected to normalization preprocessing to obtain preprocessed image tensors and segmented text fragment sets, including: Metadata is extracted and sorted from the original DICOM sequence to obtain a sorted slice stack; Three-dimensional volume data reconstruction is performed based on the sorted slice stack to obtain reconstructed 3D volume data and voxel spacing metadata; The reconstructed 3D volume data and voxel spacing metadata are subjected to 3D volume data intensity transformation and spatial normalization to obtain the preprocessed image tensor.

3. The multimodal health and medical data governance method based on a cloud platform according to claim 2, characterized in that, The process includes normalizing and preprocessing the acquired original DICOM sequence and original text report to obtain the preprocessed image tensor and the segmented text fragment set, and also includes desensitizing the original text report and semantic segmenting the original text report to obtain the segmented text fragment set.

4. The multimodal health and medical data governance method based on a cloud platform according to claim 1, characterized in that, Cross-modal feature co-encoding is performed on the preprocessed image tensor and the segmented text fragment set to obtain the image region embedding vector set and the text fragment embedding vector set, including: The preprocessed image tensor is input into a 3D convolutional neural network model pre-trained on the medical image to obtain the image region embedding vector set. The segmented text fragments are input into a language model pre-trained on a massive amount of medical text to obtain the text fragment embedding vector set.

5. The multimodal health and medical data governance method based on a cloud platform according to claim 1, characterized in that, Based on the preprocessed image tensor and the segmented text fragment set, the initial alignment mapping is calibrated and quantized using human-machine collaborative calibration to obtain the calibrated alignment mapping, including: Based on the similarity score in the initial alignment map, the alignment tasks that need to be corrected are pushed to the calibration interface. The alignment tasks that need to be corrected include text fragments and corresponding image regions. Receive interpersonal interaction results input by the doctor object; The confidence score is corrected based on the interpersonal interaction results to obtain the corrected confidence score.

6. The multimodal health and medical data governance method based on a cloud platform according to claim 5, characterized in that, The confidence score is corrected based on the interpersonal interaction results to obtain a corrected confidence score, including: correcting the confidence score based on the interpersonal interaction results using the following formula, wherein the formula is: ;in, The original similarity score for the model. For variables related to the state of human intervention, As a weighting factor, This is the corrected confidence score.

7. A multimodal health and medical data governance system based on a cloud platform, characterized in that, include: The normalization preprocessing module is used to perform normalization preprocessing on the acquired original DICOM sequence and original text report to obtain the preprocessed image tensor and the segmented text fragment set; The cross-modal embedding module is used to perform cross-modal feature co-encoding on the preprocessed image tensor and the segmented text fragment set to obtain the image region embedding vector set and the text fragment embedding vector set; The preliminary alignment module is used to perform self-supervised preliminary alignment based on contrastive learning on the image region embedding vector set and the text fragment embedding vector set to obtain the initial alignment mapping; The calibration module is used to perform human-machine collaborative calibration and confidence quantization on the initial alignment mapping based on the preprocessed image tensor and the segmented text fragment set to obtain the calibrated alignment mapping. The preliminary alignment module includes: Calculate the cosine similarity between any pair of image region embedding vectors and text fragment embedding vectors in the image region embedding vector set and the text fragment embedding vector set to obtain the cross-modal cosine similarity matrix; The method involves applying Softmax activation to the cross-modal cosine similarity matrix to obtain a set of region attention vectors. This includes: calculating the structural representation and spatial normalizing the cross-modal cosine similarity matrix to obtain a structural representation matrix, and calculating the standard deviation of the structural representation matrix to obtain the structural standard deviation; calculating the variance and standard deviation of each similarity vector in the cross-modal cosine similarity matrix; dividing the variance of each similarity vector by the structural standard deviation to obtain a specific statistical representation, and then multiplying it by the standard deviation of the corresponding similarity vector to obtain a specific variance representation; applying an exponential decay function to dynamically scale the specific variance representation to obtain a temperature scaling value; and applying Softmax activation to each similarity vector in the cross-modal cosine similarity matrix based on the temperature scaling value to obtain a set of region attention vectors. Based on the set of region attention vectors, the image region embedding vector set and the text fragment embedding vector set are used to perform optimal matching localization and initial mapping to generate the initial alignment mapping.

Images