Multi-modal fusion outpatient service data processing method and system and storage medium

Abstract

The invention discloses a multi-modal fusion outpatient service data processing method and system and a storage medium, and the method comprises the steps: obtaining patient outpatient service data of different modalities from a heterogeneous medical platform, and carrying out the desensitization processing, and obtaining the desensitization data of different modalities; carrying out feature extraction on the desensitization data of different modalities, determining different modal features and carrying out feature matching, and constructing a dynamic association map according to a matching result; generating each modal view based on the desensitization data of different modalities and rendering the modal views to an interaction page, wherein an association relationship among the modal views is defined by a dynamic association graph; and when an interactive operation of a user on any modal view is received, querying associated data of the interactive operation according to the dynamic associated graph, and synchronously rendering the associated data to each modal view. Due to the fact that the operation of the user on any view can be fed back in other views in real time, manual switching or repeated query is not needed, the diagnosis process is greatly optimized, and outpatient service efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of medical data processing technology, and in particular to a multimodal fusion outpatient data processing method, system and storage medium. Background Technology

[0002] With the deepening development of medical informatization, the data generated during outpatient diagnosis and treatment is characterized by multimodality and massive volume. Patient's medical information usually includes various types such as structured electronic medical record text, medical imaging data, and physiological time series signals. In clinical decision-making, doctors need to comprehensively refer to these scattered multi-source information to form a complete diagnostic approach.

[0003] However, because these different types of data are stored across heterogeneous platforms such as hospital information systems, image archiving systems, and laboratory systems, real-time correlation analysis is lacking. In practical applications, doctors need to frequently switch between different system pages on their local computers to view data and independently determine the correspondence between different types of data, thus affecting the efficiency of doctors' real-time diagnostic decisions. Summary of the Invention

[0004] The main purpose of this application is to provide a multimodal fusion outpatient data processing method, system and storage medium, which aims to solve the technical problem that existing outpatient data is scattered on heterogeneous platforms and lacks real-time correlation analysis.

[0005] To achieve the above objectives, this application proposes a multimodal fusion method, system, and storage medium for outpatient data processing, the method comprising: Patient outpatient data of different modalities are obtained from heterogeneous medical platforms, and the patient outpatient data of different modalities are de-identified to obtain de-identified data of different modalities. Feature extraction is performed on the desensitized data of different modalities to determine the different modal features; Feature matching is performed on the different modal features, and a dynamic association graph is constructed based on the matching results; Based on the desensitized data of different modalities, each modal view is generated and rendered to the interactive page. The relationship between each modal view is defined by the dynamic association graph. When a user interacts with an action on any of the modal views, the system queries the associated data of the interaction based on the dynamic association graph and synchronously renders the associated data to each of the modal views.

[0006] In one embodiment, the step of acquiring patient outpatient data of different modalities from a heterogeneous medical platform and performing anonymization processing on the patient outpatient data of different modalities to obtain anonymized data of different modalities includes: Data requests are sent to the heterogeneous medical platform through a standard dependency interface, so that the heterogeneous medical platform responds to the data requests by encrypting and feeding back patient outpatient data of different modalities. Upon receiving encrypted patient outpatient data of different modalities, the data is decrypted, and the decryption results are anonymized to obtain anonymized data of different modalities.

[0007] In one embodiment, the different modalities of patient outpatient data include: text data, image data, and signal data; The steps of decrypting the encrypted patient outpatient data of different modalities upon receipt and performing data anonymization processing on the decryption results to obtain anonymized data of different modalities include: Upon receiving encrypted patient outpatient data of different modalities, the data is decrypted to obtain the decrypted text data, image data, and signal data. Sensitive information is matched against the text data using regular expressions, and the matched sensitive words are replaced with mask characters to obtain de-identified text data. Sensitive metadata is identified in the image data and the signal data, and the sensitive metadata is removed from the image data and the signal data respectively to obtain desensitized image data and desensitized signal data; The desensitized text data, the desensitized image data, and the desensitized signal data are identified as desensitized data of different modalities.

[0008] In one embodiment, the desensitized data of different modalities includes: desensitized text data, desensitized image data, and desensitized signal data; Before the step of extracting features from the desensitized data of different modalities to determine the features of different modalities, the method further includes: The desensitized image data is subjected to rendering preprocessing to obtain preprocessed image data. The rendering preprocessing includes: window width and window level adjustment, noise reduction filtering, and multi-plane reconstruction. The desensitized signal data is subjected to signal correction processing to obtain processed signal data. The signal correction processing includes baseline drift correction and feature point localization. Accordingly, the step of extracting features from the desensitized data of different modalities to determine the different modal features includes: Feature extraction is performed on the desensitized text data, the preprocessed image data, and the processed signal data to obtain different modal features.

[0009] In one embodiment, the different modal features include: semantic feature vectors, spatial feature vectors, and temporal feature vectors; The step of extracting features from the desensitized text data, the preprocessed image data, and the processed signal data to obtain different modal features includes: The desensitized text data is input into a preset entity extraction model to obtain the current text entity, and the current text entity is converted into a semantic feature vector, which is obtained by training with sample text data and sample text entities; The preprocessed image data is input into a preset image segmentation model to obtain a current lesion segmentation mask, and the current lesion segmentation mask is converted into a spatial feature vector. The preset image segmentation model is trained using sample image data and sample lesion segmentation masks. An abnormal signal segment is identified from the processed signal data using a dynamic time warping algorithm, and the abnormal signal segment is converted into a time-series feature vector.

[0010] In one embodiment, the step of performing feature matching on the different modal features and constructing a dynamic association graph based on the matching results includes: Calculate the corresponding attention weights among the semantic feature vector, the spatial feature vector, and the temporal feature vector, respectively; A plurality of graph nodes are generated based on the semantic feature vector, the spatial feature vector and the temporal feature vector, the graph nodes including: text nodes, image nodes and signal nodes; Based on the corresponding attention weights, the connection relationships between the graph nodes are determined, and several connection edges are generated. By integrating all the graph nodes and all the connecting edges, a dynamic association graph is constructed.

[0011] In one embodiment, the step of querying the associated data of the interaction based on the dynamic association graph and synchronously rendering the associated data to each of the modal views when receiving an interaction operation from a user on any of the modal views includes: When a user interacts with the current modal view, the first graph node corresponding to the interaction is determined, wherein the current modal view is a text view, an image view, or a signal view. The dynamic association graph queries other graph nodes that are connected to the first graph node, and the first graph node and the other graph nodes belong to different modal views; The first associated modal data corresponding to the first graph node is rendered to the current modal view, and the other associated modal data corresponding to the other graph nodes are rendered to other modal views.

[0012] In one embodiment, after the step of synchronously rendering the associated data to each of the modal views, the method further includes: Obtain the updated modal view and receive feedback from the user on the updated modal view; The dynamic association graph is updated according to the feedback operation, and the association relationships between the modal views are redefined based on the updated dynamic association graph.

[0013] In addition, to achieve the above objectives, this application also proposes a multimodal fusion outpatient data processing system, the system comprising: a front-end device and a back-end server, the back-end server being equipped with a heterogeneous medical platform, and the front-end device comprising: a data acquisition module, a front-end preprocessing module, a multimodal fusion module, and a rendering and interaction module; The data acquisition module is used to acquire patient outpatient data of different modalities from the heterogeneous medical platform; The front-end preprocessing module is used to desensitize the patient outpatient data of different modalities to obtain desensitized data of different modalities. The multimodal fusion module is used to extract features from the desensitized data of different modalities to determine different modal features. The multimodal fusion module is also used to perform feature matching on the different modal features and construct a dynamic association map based on the matching results; The rendering interaction module is used to generate modal views based on the desensitized data of different modalities and render them to the interactive page. The relationship between the modal views is defined by the dynamic association graph. The rendering interaction module is further configured to, upon receiving an interaction operation from a user on any of the modal views, query the associated data of the interaction operation based on the dynamic association graph, and synchronously render the associated data to each of the modal views.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a multimodal fusion outpatient data processing program is stored, wherein when the multimodal fusion outpatient data processing program is executed by a processor, it implements the steps of the multimodal fusion outpatient data processing method described above.

[0015] This application discloses a multimodal fusion outpatient data processing method, comprising: acquiring patient outpatient data of different modalities from a heterogeneous medical platform, and desensitizing the patient outpatient data of different modalities to obtain desensitized data of different modalities; extracting features from the desensitized data of different modalities to determine different modal features; performing feature matching on the different modal features and constructing a dynamic association graph based on the matching results; generating each modal view based on the desensitized data of different modalities and rendering it to an interactive page, wherein the association relationship between each modal view is defined by the dynamic association graph; and when receiving an interactive operation from a user on any modal view, querying the associated data of the interactive operation according to the dynamic association graph and synchronously rendering the associated data to each modal view.

[0016] Because this application can desensitize and extract features from multimodal outpatient data, and construct a dynamic association graph to define the relationship between each modal view, it can respond to and update each modal view in real time through the dynamic association graph. This allows users to receive immediate feedback in other views on any operation in any view without having to manually switch or repeatedly query, which greatly optimizes the diagnostic process and improves outpatient efficiency. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the outpatient data processing method for multimodal fusion according to this application; Figure 2 This is a flowchart illustrating the second embodiment of the outpatient data processing method for multimodal fusion according to this application; Figure 3 This is a schematic diagram of the desensitization process for multi-source encrypted data. Figure 4 A schematic diagram of the computational acceleration preprocessing flow for multi-source de-identified data; Figure 5 This is a flowchart illustrating the third embodiment of the outpatient data processing method for multimodal fusion according to this application; Figure 6 A schematic diagram of the feature extraction process for multimodal input data; Figure 7 A schematic diagram illustrating the feature association process for different modalities; Figure 8 This is a schematic diagram of the multimodal view-associated rendering process; Figure 9 A schematic diagram illustrating the entire process of outpatient data processing using multimodal fusion. Figure 10 This is a schematic diagram of the outpatient data processing system based on multimodal fusion in this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] This application provides a multimodal fusion method for outpatient data processing, referencing... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the outpatient data processing method for multimodal fusion according to this application. In this embodiment, the method includes steps S10 to S50: Step S10: Obtain patient outpatient data of different modalities from the heterogeneous medical platform, and perform desensitization processing on the patient outpatient data of different modalities to obtain desensitized data of different modalities.

[0024] It should be noted that the method in this embodiment can be applied in medical diagnostic scenarios to assist doctors in making diagnostic decisions. The executing entity of this embodiment can be a front-end module of a computing electronic device with data processing, network communication, and browser page display functions, such as a web browser running on a doctor's workstation computer; it can also be an additionally deployed edge gateway device, which can integrate a Digital Imaging and Communications in Medicine (DICOM) decoder and a lightweight AI model, and can achieve real-time communication with the front-end module via WebSocket. The following embodiments are described using a web browser running on a doctor's workstation computer (referred to as the "front-end device") as an example.

[0025] It should be understood that heterogeneous healthcare platforms can be set up on backend servers. These platforms can be healthcare systems that store different modalities of medical data, such as Hospital Information Systems (HIS, used to obtain text medical records), Picture Archiving Systems (PACS, used to obtain DICOM images), and Laboratory Systems (LIS, used to obtain physiological signals). These different systems are independent of each other, and front-end devices can request patient outpatient data of different modalities in parallel through standard medical information exchange interfaces (such as FHIR / HL7).

[0026] Patient outpatient data in different modalities can include: text data, image data, and signal data. To prevent the leakage of patient privacy, the above-mentioned patient outpatient data in different modalities can be anonymized at the underlying level of the front-end device (such as within the browser sandbox): for text data, use regular expressions to match and mask sensitive information such as ID card numbers and phone numbers; for image and signal data, directly prevent sensitive metadata such as patient names and device IDs from being transmitted and displayed to the upper-level interactive interface of the front-end device.

[0027] In addition, to facilitate subsequent multimodal data synchronization, spatiotemporal identifiers can be assigned to the patient outpatient data of the above different modalities: paragraph position coordinates are marked for text data, three-dimensional spatial coordinates (x, y, z and time axis) are labeled for image data, and time axis intervals are bound to signal data.

[0028] Step S20: Extract features from the desensitized data of different modalities to determine the different modal features.

[0029] It should be understood that front-end devices can run lightweight AI models that can extract features from data of different modalities, such as the BERT text model and the MobileNetV3 segmentation model.

[0030] By inputting desensitized data of different modalities into corresponding lightweight AI models for feature extraction, or by using different feature extraction algorithms, desensitized data of different modalities can be converted into feature vectors of the same dimension.

[0031] In specific implementation, entity feature extraction can be performed on desensitized text data to obtain text entity information, including disease entities and their corresponding descriptive paragraphs; lesion detection can be performed on desensitized image data to detect different lesions and mark them on the image using bounding boxes, where the lesion is the image representation of the disease entity; and abnormal signal fragment identification can be performed on desensitized signal data to obtain abnormal signal fragments.

[0032] Step S30: Perform feature matching on the different modal features and construct a dynamic association map based on the matching results.

[0033] It should be understood that different modal features can first be mapped to the same feature space, for example, by using vectors to represent them, to obtain semantic feature vectors, spatial feature vectors, and temporal feature vectors.

[0034] For the three different types of feature vectors mentioned above, an attention fusion mechanism can be used to match one type of feature vector as a query with another type of feature vector. For example, the text feature "right lung nodule" can be used as a query to match with the image feature, thereby triggering the image model to perform high-precision detection in the right lung region; conversely, the image feature (detected nodules) can also be matched back to the text paragraphs in the text data that describe "nodule diameter > 5mm".

[0035] It is understandable that the above matching results can be constructed into a dynamic association graph, which can consist of several nodes and connecting edges. Nodes can be text entities, image lesions, or signal anomalies; edges can be spatial co-occurrence (same location), temporal synchronization (same time), semantic relevance (describes the same thing) and other associations between the above nodes.

[0036] Step S40: Generate modal views based on the desensitized data of different modalities and render them to the interactive page. The relationship between the modal views is defined by the dynamic association graph.

[0037] It should be noted that the modal views can be text views, image views, and signal views. Text views can be structured electronic medical record text rendered in a webpage; image views can include 3D views of CT / MRI generated in real time using the WebGPU volume renderer; signal views can be physiological signal waveforms such as electrocardiograms (ECG) drawn based on a time axis.

[0038] It should be understood that desensitized image data can be several CT / MRI image slice sequences. Therefore, CT / MRI image slice sequences can be input into the WebGPU volume renderer to generate a 3D image view of CT / MRI in real time through a ray projection algorithm. It also supports interactive operations such as 3D dynamic cutting and transparency adjustment based on different lesions.

[0039] It should also be noted that the mapping relationship between each screen coordinate in the different modal views and each node in the dynamic association graph can be established in advance, and the names of the nodes with the mapping relationship can be directly displayed at each screen coordinate in different modal views.

[0040] In addition, the dynamic association graph can also be used as a logical rule to define the association relationship between screen coordinates (node ​​names) belonging to different modal views. For example, the dynamic association graph can define the association between "node A" and "text paragraph B" and "time point C".

[0041] Step S50: Upon receiving an interactive operation from a user on any of the modal views, query the associated data of the interactive operation according to the dynamic association graph, and synchronously render the associated data to each of the modal views.

[0042] For example, a user (doctor) can select a lesion (nodule A) on the image view to generate a click event. The click event can include the name of the lesion and the coordinates of the bounding box. Then, based on the click event, all the associated data in the dynamic association map can be queried, including the associated text description paragraphs and abnormal signal segments associated with the lesion.

[0043] Next, the front-end device can calculate and display information such as nodule volume in real time in the image view, automatically scroll and highlight the queried text paragraphs in the text view, and automatically position the time axis to the time interval corresponding to the abnormal segment in the signal view, synchronously playing the auscultation recording to that time interval. This enables the front-end device to drive the synchronous updates and rendering of each modal view based on the associated data, making the relationships between the modal views clearer and facilitating comprehensive analysis and judgment by the user.

[0044] Furthermore, in order to optimize the relationship between the modal views, after step S50, steps S601 to S603 are also included.

[0045] Step S601: Obtain the updated modal view and receive feedback from the user on the updated modal view.

[0046] For example, if the front-end device automatically associates the text "liver cyst" with a shaded area on an image based on a dynamic association map, and the user, upon seeing this association in the updated modal view, judges based on their own experience that the association is incorrect (the text corresponding to the shaded area is "cholecystitis"), then the user can manually drag the text paragraph description corresponding to "liver cyst" to place it in the correct gallbladder area on the image view.

[0047] In practical implementation, the front-end device can record the corrective actions based on the user feedback mentioned above and generate feedback data, which may include correction vectors (e.g.: x, y, z (the coordinate offset from the incorrect position to the correct position) and the corresponding data context.

[0048] Step S602: Update the dynamic association graph according to the feedback operation, and redefine the association relationship between each modal view based on the updated dynamic association graph.

[0049] It should be understood that the front-end device can use the above feedback data to update the dynamic association map. For example, in the dynamic association map, the association edge between "liver cyst" and the above shaded area can be weakened or deleted, while the association edge between "liver cyst" and the correct gallbladder area can be established or strengthened.

[0050] Furthermore, front-end devices can periodically upload large amounts of similar feedback data (containing only correction vectors and excluding sensitive patient information) to the cloud. The cloud then aggregates the correction data from users on different front-end devices and distributes it to the lightweight AI model on the front-end device for model optimization, thereby further improving the correlation accuracy of different modal views.

[0051] This embodiment can desensitize and extract features from multimodal outpatient data, and construct a dynamic association graph to define the relationship between each modal view. This allows the dynamic association graph to respond to and update each modal view in real time, so that the user's operation on any view can be immediately reflected in other views without the need for manual switching or repeated queries, which greatly optimizes the diagnostic process and improves outpatient efficiency.

[0052] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the outpatient data processing method for multimodal fusion according to this application.

[0053] In this embodiment, to further illustrate how data anonymization is achieved, step S10 includes: steps S101~S102: Step S101: Send a data request to the heterogeneous medical platform through the standard dependency interface, so that the heterogeneous medical platform responds to the data request by encrypting and feeding back patient outpatient data of different modalities.

[0054] It should be noted that different medical platforms (HIS, PACS, LIS) can each have their own corresponding backend servers, which store the medical data specific to that platform. Frontend devices can communicate with different medical platforms via interfaces such as FHIR and HL7.

[0055] It should also be noted that when the system's backend server receives a data request from the frontend device, it can also encrypt the patient data of different modalities to be transmitted with a key, package it into an encrypted data packet, and then send it, thereby preventing the leakage of patient data.

[0056] Step S102: Upon receiving the encrypted patient outpatient data of different modalities, decrypt the data and perform data anonymization processing on the decryption results to obtain anonymized data of different modalities.

[0057] It should be understood that after receiving encrypted data packets transmitted from the backend servers of various systems, the front-end device can decrypt them and recover the original patient data in different modalities. This original patient data can reside in the front-end device's memory or underlying modules, and has not yet been used or displayed, thereby further preventing patient data leakage.

[0058] Next, the front-end device can perform data anonymization processing on the above-mentioned patient outpatient data (text data, image data, and signal data) of different modalities to obtain anonymized data of different modalities.

[0059] The text data can be electronic medical records from the HIS system, including patient complaints and medical history; the image data can be DICOM format images from the PACS system, including CT images, MRI images, etc.; and the signal data can be physiological time-series signals from the LIS, including ECG signals, blood oxygenation signals, etc.

[0060] It should also be noted that when receiving image data (DICOM format image), in order to facilitate subsequent operations, a DICOM decoder compiled by WebAssembly can be used to directly convert the binary image data into an operable pixel matrix in the browser, avoiding the time-consuming problem of backend server decoding required in traditional solutions.

[0061] Therefore, to specifically illustrate how to desensitize the patient outpatient data of the above different modalities, step S102 includes: steps S1021~S1024: Step S1021: Upon receiving the encrypted patient outpatient data of different modalities, decrypt the data to obtain the decrypted text data, image data, and signal data.

[0062] In practice, the front-end device decrypts the encrypted data packet upon receiving it to obtain the decrypted text data, image data, and signal data.

[0063] Step S1022: Use regular expressions to match sensitive information in the text data, and replace the matched sensitive words with mask characters to obtain de-sensitized text data.

[0064] It should be understood that sensitive information may include sensitive words predefined by the user in the front-end device, such as "patient name:" or predefined sensitive word patterns, such as specific combinations of numbers from an ID card or phone number.

[0065] In addition, sensitive information can be categorized into direct and indirect identifiers. Direct identifiers can include: name, ID number, phone number, and address; indirect identifiers can include: date of visit, doctor's name, and department information.

[0066] In practice, regular expressions can be used to match sensitive information in text data, directly replacing sensitive words with asterisks (*) or mask characters such as "[already desensitized]" to obtain desensitized text data. This desensitized text data retains the original medical content information but removes patient identifiers, helping to protect patient privacy.

[0067] For example, the original text before desensitization was: "Patient Zhang San (ID card number 110101200001011234) visited Dr. Li Si, the head of the respiratory department, on January 1, 2020 due to a nodule in his right lung." The anonymized text would then read: "Patient *** (ID card number 110101********1234) visited the outpatient clinic of [Department] [Doctor] on [Date] due to a nodule in the right lung."

[0068] Step S1023: Identify sensitive metadata in the image data and the signal data, and remove the sensitive metadata from the image data and the signal data respectively to obtain desensitized image data and desensitized signal data.

[0069] It should be understood that, in addition to pixel images and signal waveforms, image data and signal data also contain their corresponding metadata header files, which record sensitive information such as patient name, hospital name, medical equipment number, and examination date.

[0070] The pixel image may include the patient’s facial features (e.g., craniofacial CT), implant identification, etc.; the signal waveform may include voiceprint features from voice consultation.

[0071] In practice, the front-end device can directly process the aforementioned sensitive metadata in the GPU memory, removing it from the image data and signal data respectively, and only processing privacy-sensitive areas (such as the face) while preserving the integrity of the diagnostic area, thereby obtaining desensitized image data and desensitized signal data. Furthermore, since the sensitive metadata does not enter the display pipeline, patient privacy can be further protected.

[0072] Step S1024: Desensitize the desensitized text data, the desensitized image data, and the desensitized signal data as desensitized data of different modalities.

[0073] It should be understood that the front-end device de-identifies all patient outpatient data from different modalities in the back-end, generating safe, compliant, and fully identifiable de-identified data that retains all medical content. This data can then serve as the foundation for subsequent feature extraction and modal data fusion analysis.

[0074] Furthermore, since encrypted patient outpatient data in different modalities (i.e., multi-source encrypted data) can be directly input into the front-end desensitization engine for desensitization processing, this can be referenced here. Figure 3 The process of desensitizing the aforementioned multi-source encrypted data will be explained in a unified manner. Figure 3 This is a schematic diagram of the desensitization process for multi-source encrypted data.

[0075] Depend on Figure 3 It can be seen that the desensitization of multi-source encrypted data can be divided into text desensitization, image desensitization, and signal desensitization.

[0076] In text desensitization, regular expression matching masks and NER entity replacement can be used to desensitize the original text.

[0077] In image desensitization, metadata erasure and pixel-level processing methods can be used to desensitize the original image data.

[0078] In signal desensitization, methods such as using identification glass and retaining feature values ​​can be used to desensitize the original signal data.

[0079] The de-identified data can be stored in the secure memory area of ​​the front-end device for subsequent AI analysis.

[0080] Furthermore, before extracting features of different modalities, in order to improve the accuracy and effectiveness of subsequent feature extraction, the de-identified data can be preprocessed. Therefore, before step S20, steps S01~S02 are also included: Step S01: Perform rendering preprocessing on the desensitized image data to obtain preprocessed image data. The rendering preprocessing includes: window width and window level adjustment, noise reduction filtering, and multi-plane reconstruction.

[0081] It should be noted that window width and window level adjustments are a way to adjust medical contrast and brightness. Since DICOM format images are usually in grayscale format, and certain pathological features are only visible within a specific grayscale range, adjusting the window width and window level can highlight specific tissues (such as bones and soft tissues) in DICOM format images.

[0082] Anisotropic noise reduction filtering algorithms that preserve edge features can be selected to enhance useful information in the image while suppressing random noise and improving the signal-to-noise ratio. This allows subsequent models to more accurately identify lesions and avoid interference from noise.

[0083] Since CT and MRI images are a series of axial slices, they can be reconstructed in multiple planes using computational shaders, which can generate views from other angles such as the coronal and sagittal planes in real time, thereby enriching the spatial features in the image data.

[0084] Step S02: Perform signal correction processing on the desensitized signal data to obtain processed signal data. The signal correction processing includes baseline drift correction and feature point localization.

[0085] It should be understood that during the acquisition of physiological signals (such as ECG), factors such as patient movement may cause the entire signal waveform to drift vertically. Therefore, baseline drift correction can flatten the baseline of the signal waveform, bringing it back to the standard position, eliminating the influence of vertical drift, and thus ensuring the accuracy of the signal data.

[0086] In addition, to further improve the efficiency of subsequent signal feature extraction, taking ECG as an example, a peak detection algorithm can be used to extract key feature points (such as R wave, P wave, T wave) in each heartbeat cycle of the ECG signal waveform and annotate them in the aforementioned desensitized signal data.

[0087] Accordingly, step S20 specifically includes: Step S200: Extract features from the desensitized text data, the preprocessed image data, and the processed signal data to obtain different modal features.

[0088] It should be understood that the accuracy of the obtained modal features is ensured because the anonymized data of different modalities were preprocessed before feature extraction.

[0089] Furthermore, you can also refer to this section. Figure 4 This application provides a unified description of the preprocessing procedures for de-identified data of different modalities (multi-source de-identified data). Figure 4 This is a schematic diagram of the calculation acceleration preprocessing flow for multi-source de-identified data.

[0090] Depend on Figure 4It is known that for anonymized text data, the anonymized medical text can be converted into numerical feature vectors: text data is converted into vector form based on word vectorization Web Worker technology, which facilitates subsequent processing; and entities in the text are encoded based on entity encoding (WebAssembly, WASM) technology. This eliminates lexical variations while preserving clinical semantics.

[0091] For de-identified image data, Web GPUs can be used for texture compression to convert the de-identified DICOM pixel data into standardized feature maps, thereby extracting multi-scale image features to support subsequent AI analysis. Furthermore, computational shaders can be used to perform 3D reconstruction of the de-identified DICOM pixel data to enrich the spatial features within the image data.

[0092] For desensitized signal data, the WebAudio API can be used to filter it, and SIMD (Single Instruction Multiple Data) technology can be used for feature extraction to improve processing efficiency.

[0093] Finally, the processing results of the above three data types can be integrated into a multimodal feature matrix to facilitate subsequent multimodal data matching.

[0094] This embodiment reduces the computational load on the backend server by constructing a dual security system of transmission encryption and front-end desensitization. Different desensitization processing is applied to different modal data in the front-end device, thereby maximizing the preservation of medical content in the original data while protecting patient privacy. Subsequently, the desensitized data undergoes computational acceleration processing, providing reliable and clean input data for subsequent feature extraction and association analysis. This is beneficial for improving the accuracy of the extracted modal features and the reliability of the subsequently constructed dynamic association map.

[0095] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the outpatient data processing method for multimodal fusion of this application.

[0096] In this embodiment, to specifically illustrate how to extract features of different modalities, step S200 specifically includes: steps S201~S203: Step S201: Input the desensitized text data into a preset entity extraction model to obtain the current text entity, and convert the current text entity into a semantic feature vector.

[0097] It should be understood that the preset entity extraction model is a model trained using sample text data and sample text entities, such as a fine-tuned BERT model.

[0098] It should be noted that the above-mentioned preset entity model can identify key clinical entities from the anonymized medical record text, such as the disease entity "gastric antral ulcer" and map it to the Standard Medical Terminology Database (UMLS); or the anatomical location "right middle lobe of the lung" and convert it into three-dimensional spatial coordinates.

[0099] In the specific implementation, after identifying several clinical entities as the current text entities based on the preset entity extraction model, a semantic feature vector can be constructed by combining the positional encoding of each clinical entity in the text data.

[0100] Step S202: Input the preprocessed image data into a preset image segmentation model to obtain the current lesion segmentation mask, and convert the current lesion segmentation mask into a spatial feature vector.

[0101] It should be understood that the preset image segmentation model is a model trained using sample image data and sample lesion segmentation masks, such as the MobileNetV3 model based on TensorFlow.js.

[0102] It should be noted that the aforementioned preset image segmentation model can detect different lesions (different lesions can be named different disease entities) from the preprocessed DICOM format image and output different lesion segmentation masks. Based on the lesion segmentation mask, the three-dimensional bounding box (x, y, z, length, width, height) of different lesions can be determined, and then the specific spatial information (length, width, height) can be encoded into a spatial feature vector.

[0103] Step S203: Use a dynamic time warping algorithm to determine abnormal signal segments from the processed signal data, and convert the abnormal signal segments into time-series feature vectors.

[0104] It should be understood that Dynamic Time Warping (DTW) is a time-series data matching algorithm that can flexibly align two waveforms that have slightly different speeds.

[0105] Specifically, based on DTW, the processed ECG signal can be compared with a normal signal template, and abnormal segments (such as a segment of "ventricular fibrillation" or "tachycardia") can be identified by calculating the shortest path; then the start time, duration, abnormal type and other information of the abnormal segment are encoded into the corresponding time-series feature vector.

[0106] In addition, you can refer to this place. Figure 6 The extraction process of the above-mentioned different modal features is integrated and explained. Figure 6 This is a schematic diagram of the feature extraction process for multimodal input data.

[0107] Depend on Figure 6 As can be seen, when extracting text features, the BERT model can be used to fine-tune the text data to adapt to specific medical text processing tasks. Then, medical entity recognition technology is used to identify medical-related entities in the text. Next, attention mechanism is used to pool the text features and extract key information. Finally, feature vectors representing the semantics of the text data can be generated.

[0108] When extracting image features, we can first use a 3D convolutional neural network to process the image data and extract spatial features; then we can perform feature pooling on the region of interest (ROI) to extract key features; then we can extract features at different levels through multi-scale processing; finally, we can generate feature vectors that represent the spatial features of the image data.

[0109] When extracting signal features, the signal data can first be analyzed in terms of time and frequency to extract frequency and time features; then, the morphological features of the signal, such as shape and structure, can be extracted; then, the dynamic time warping (DTW) algorithm can be used to process the time series features of the signal; finally, feature vectors representing the time series features of the signal data can be generated.

[0110] Finally, the different modal features mentioned above can be standardized to ensure that they are on a uniform scale, which facilitates subsequent feature association and fusion.

[0111] Furthermore, to specifically illustrate how to perform feature association on the above-mentioned different modal features, step S30 includes: steps S301~S304: Step S301: Calculate the corresponding attention weights among the semantic feature vector, the spatial feature vector, and the temporal feature vector.

[0112] It should be understood that attention weights between different modalities can be calculated separately, including: text-image attention weights, image-text attention weights, temporal-text / image attention weights, and text / image-temporal attention weights.

[0113] The text-image attention weight can be calculated using semantic feature vectors as queries and spatial feature vectors as keys and values. For example, the semantic feature vector of "right lung nodule" can be used to query all spatial feature vectors and calculate a weight score, which represents the degree of correlation between "right lung nodule" and each lesion region in the image.

[0114] Image-text attention weights can be calculated using spatial feature vectors as queries and semantic feature vectors as keys and values. For example, the spatial feature vector of "right lung nodule" can be used to query all semantic feature vectors, and a weight score can be calculated, which represents the relevance of "right lung nodule" to each text entity in the text.

[0115] Temporal-text / image attention weights can be calculated using temporal feature vectors as queries, with semantic and spatial feature vectors as keys and values, respectively. For example, the temporal feature vector of "tachycardia" can be used to query all semantic / spatial feature vectors to calculate a weight score, which represents the degree of correlation between "tachycardia" and each text entity in the text or each lesion region in the image.

[0116] The interpretation of text / image-temporal attention weights is similar to that of text-image attention weights, image-text attention weights, and temporal-text / image attention weights, and will not be elaborated upon in this embodiment.

[0117] Step S302: Generate several graph nodes based on the semantic feature vector, the spatial feature vector, and the temporal feature vector. The graph nodes include: text nodes, image nodes, and signal nodes.

[0118] It should be understood that each modal feature extracted above can be transformed into a node in the atlas, where a text node can represent a clinical entity (e.g., "right lung nodule", "diameter > 5 mm"); an image node can represent a lesion region on an image (defined by its spatial feature vector); and a signal node can represent an abnormal signal segment (e.g., "tachycardia zone").

[0119] Step S303: Determine the connection relationship between each graph node according to the corresponding attention weight, and generate several connection edges.

[0120] It should be noted that an attention weight threshold can be preset. If the attention weight between two graph nodes exceeds the threshold, it can be considered that there is a connection edge between the two graph nodes, and the corresponding weight of the connection edge is the aforementioned attention weight.

[0121] It should also be noted that the types of each connecting edge can include: semantic-spatial association edges, temporal-semantic association edges, and spatial-temporal association edges.

[0122] For example, a semantic-spatial association edge can be an edge between the text node "right lung nodule" and the image node "nodule region"; a temporal-semantic association edge can be an edge between the signal node "tachycardia" and the text node "complaint of palpitation"; and a spatial-temporal association edge can be an edge between the image node "lesion A" and the signal node "abnormal ECG during examination".

[0123] Step S304: Integrate all the graph nodes and all the connecting edges to construct a dynamic association graph.

[0124] In the specific implementation, all nodes and all edges are gathered together to form a complete graph-structured data model, namely a dynamic relational graph, which can be used to respond to user interaction operations in real time.

[0125] Accordingly, step S40 includes: initializing each modal view based on desensitized data of different modalities, establishing a mapping relationship between different screen coordinates in each modal view and each graph node, and determining each modal view that needs to be rendered to the interactive interface based on each mapping relationship and the initialized modal views.

[0126] In addition, you can refer to this place. Figure 7 The process of feature association for different modal features is integrated and explained. Figure 7 This is a schematic diagram of the feature association process for different modal features.

[0127] Depend on Figure 7 It can be seen that when different modal features (semantic feature vectors, spatial feature vectors, and temporal feature vectors) are obtained, feature association can be performed by the feature matching engine.

[0128] First, spatial similarity, temporal similarity, and semantic similarity between each pair of different modalities can be calculated based on spatial distance measurement, temporal analysis, and semantic similarity calculation. Spatial weights, temporal weights, and semantic weights corresponding to spatial similarity, temporal similarity, and semantic similarity are assigned according to pre-set similarity weights. The comprehensive correlation between features of different modalities is obtained based on the assigned weights and the corresponding similarities.

[0129] Next, spatial co-occurrence association, temporal synchronization association, and semantic association can be achieved based on the comprehensive correlation between the above-mentioned different modal features.

[0130] Finally, several graph nodes can be generated based on the above different modal features, and the edge connection relationships between each graph node can be determined according to the above associations, thereby constructing a dynamic association graph.

[0131] In addition, this dynamic association graph can be visualized in real time, allowing users to explore it interactively.

[0132] Furthermore, in order to achieve multimodal view-related rendering, step S50 specifically includes: steps S501~S503: Step S501: When receiving an interactive operation from the user on the current modal view, determine the first graph node corresponding to the interactive operation, wherein the current modal view is a text view, an image view, or a signal view.

[0133] It should be understood that this interactive operation can be an operation in which a user selects a segment of medical record text in the text view, a user selects a segment of image in the image view, or a user drags a segment on the timeline in the signal view; the node pointed to by the user, i.e., the first atlas node, is determined according to the position of the different interactive operations on the view.

[0134] For example, if a user clicks on a lung nodule in the image view, the screen coordinates of the user's click can be mapped to image node A representing that nodule in the image view, and that image node A is the first atlas node.

[0135] Step S502: Query other graph nodes that are connected to the first graph node through the dynamic association graph, wherein the first graph node and the other graph nodes belong to different modal views.

[0136] It should be understood that, with the first graph node as the center, the dynamic association graph can be traversed to find other graph nodes that have connecting edges with the first graph node. Then, the other graph node has an association relationship with the first graph node and belongs to other modal views other than the current modal view.

[0137] For example, if the first atlas node is an image node A, then other atlas nodes can be text nodes B connected to the image node A, which correspond to the medical record segment describing the image node A; or they can be signal nodes C connected to the image node A, which correspond to the electrocardiogram abnormality segment recorded synchronously when the image was acquired.

[0138] Step S503: Render the first associated modal data corresponding to the first graph node to the current modal view, and render the other associated modal data corresponding to the other graph nodes to other modal views.

[0139] It should be understood that the first associated modal data can be the data represented by the first graph node itself, which needs to be enhanced and displayed on the current modal view. It can be a direct feedback of the user's interactive operations on the current modal view. For example, highlighting a bounding box around the node clicked by the user, or calculating and displaying its volume in real time.

[0140] Other related modal data can be data extracted from other graph nodes that needs to be displayed on other modal views, including text paragraph IDs that need to be highlighted and time points that need to be located.

[0141] In the specific implementation, in response to user interaction, data enhancement is performed on different modal views, including: highlighting the image node A clicked by the user and displaying its corresponding volume in the image view; automatically scrolling to the paragraph corresponding to text node B in the text view and highlighting it; and automatically positioning and zooming the time axis to the abnormal time period corresponding to signal node C in the signal view; thus realizing cross-modal view linkage.

[0142] In addition, you can refer to this place. Figure 8 The process of rendering multimodal views in association described above will be integrated and explained. Figure 8 This is a schematic diagram of the multimodal view-associated rendering process.

[0143] Depend on Figure 8 It can be seen that the front-end device can respond to the user's interactive operation based on the interactive event resolver, thereby querying the associated graph nodes in the graph, determining the associated modal data in different modal views, and rendering them accordingly.

[0144] Specifically, the text view renderer can perform DOM manipulation and highlight relevant text; the image view renderer can use WebGPU for real-time rendering; and the signal view renderer can use Canvas for waveform drawing.

[0145] Based on the aforementioned interactive event parser, the unified state of all views can be managed to ensure data consistency; and the feedback loop of user operations can be detected to ensure the responsiveness and stability of the system.

[0146] In addition, the front-end device can also record the view rendering speed, response time, etc., based on the above process, which helps to further optimize interaction latency and improve user experience.

[0147] This embodiment first uses different feature extraction methods to transform unstructured text, images, and signals into structured, computable feature vectors. Then, it uses an attention mechanism to determine the correlation between feature vectors of different modalities to construct a dynamic correlation graph. This helps to build cross-modal view correlations, allowing users to trigger collaborative responses in other modal views through interactive operations in one modal view. This facilitates users to quickly verify diagnostic hypotheses based on different modal data. Compared to the traditional method of manually searching and associating cross-modal data, which heavily relies on doctors' experience and short-term memory, this embodiment ensures the objectivity and completeness of the correlation response through the dynamic correlation graph, helping to provide a more reliable data foundation for user decision-making.

[0148] Furthermore, you can also refer to this section. Figure 9 This paper describes the entire process of the outpatient data processing method based on multimodal fusion in this application. Figure 9 This is a schematic diagram of the entire process of outpatient data processing using multimodal fusion.

[0149] First, the front-end device can collect patient outpatient data of different modalities from the back-end server of the heterogeneous medical platform and perform front-end preprocessing, including privacy desensitization processing and computation acceleration processing (rendering preprocessing for image data and signal correction processing for signal data). Next, feature extraction is performed on the processed modal data to obtain different modal features, and then cross-modal feature association is performed to construct a dynamic association map; Finally, each modal data is rendered into a corresponding modal view, and the relationship between each modal view is defined through a dynamic association graph, thereby responding to user interaction operations by rendering associated data on each modal view.

[0150] Furthermore, after obtaining the updated modal view, the dynamic association map is dynamically updated based on user feedback, thereby further optimizing the association relationship between the various modal views.

[0151] This application also proposes a multimodal fusion outpatient data processing system, referencing... Figure 10 , Figure 10 This is a schematic diagram of the outpatient data processing system based on multimodal fusion in this application.

[0152] Depend on Figure 10 It is known that the system includes: a front-end device 10 and a back-end server 20. The back-end server 20 is equipped with a heterogeneous medical platform. The front-end device 10 includes: a data acquisition module 101, a front-end preprocessing module 102, a multimodal fusion module 103, and a rendering and interaction module 104.

[0153] The data acquisition module 101 is used to acquire patient outpatient data of different modalities from the heterogeneous medical platform; The front-end preprocessing module 102 is used to desensitize the patient outpatient data of different modalities to obtain desensitized data of different modalities. The multimodal fusion module 103 is used to extract features from the desensitized data of different modalities to determine different modal features; The multimodal fusion module 103 is also used to perform feature matching on the different modal features and construct a dynamic association map based on the matching results; The rendering interaction module 104 is used to generate each modal view based on the desensitized data of different modalities and render it to the interactive page. The relationship between each modal view is defined by the dynamic association graph. The rendering interaction module 104 is further configured to, upon receiving an interaction operation from a user on any of the modal views, query the associated data of the interaction operation based on the dynamic association graph, and synchronously render the associated data to each of the modal views.

[0154] This embodiment desensitizes and extracts features from multimodal outpatient data, constructs a dynamic association graph to define the relationships between various modal views, and enables real-time response and synchronous updates to each modal view through the dynamic association graph. This allows users to receive immediate feedback in other views for their actions on any view without the need for manual switching or repeated queries, greatly optimizing the diagnostic process and improving outpatient efficiency.

[0155] This application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the outpatient data processing method of multimodal fusion in the above embodiments.

[0156] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0157] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multimodal fusion outpatient data processing method, thereby solving the technical problems of the multimodal fusion outpatient data processing method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the multimodal fusion outpatient data processing method provided in the above embodiments, and will not be repeated here.

[0158] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other elements in the process, method, article, or system that includes that element.

[0159] The sequence numbers of the above embodiments of the present invention are merely for description and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and are not intended to limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and based on the content of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A method for processing outpatient data using multimodal fusion, characterized in that, The method includes: Patient outpatient data of different modalities are obtained from heterogeneous medical platforms, and the patient outpatient data of different modalities are de-identified to obtain de-identified data of different modalities. Feature extraction is performed on the desensitized data of different modalities to determine the different modal features; Feature matching is performed on the different modal features, and a dynamic association graph is constructed based on the matching results; Based on the desensitized data of different modalities, each modal view is generated and rendered to the interactive page. The relationship between each modal view is defined by the dynamic association graph. When a user interacts with an action on any of the modal views, the system queries the associated data of the interaction based on the dynamic association graph and synchronously renders the associated data to each of the modal views.

2. The method as described in claim 1, characterized in that, The steps of acquiring patient outpatient data of different modalities from a heterogeneous medical platform and de-identifying the patient outpatient data of different modalities to obtain de-identified data of different modalities include: Data requests are sent to the heterogeneous medical platform through a standard dependency interface, so that the heterogeneous medical platform responds to the data requests by encrypting and feeding back patient outpatient data of different modalities. Upon receiving encrypted patient outpatient data of different modalities, the data is decrypted, and the decryption results are anonymized to obtain anonymized data of different modalities.

3. The method as described in claim 2, characterized in that, The patient outpatient data in different modalities includes: text data, image data, and signal data; The steps of decrypting the encrypted patient outpatient data of different modalities upon receipt and performing data anonymization processing on the decryption results to obtain anonymized data of different modalities include: Upon receiving encrypted patient outpatient data of different modalities, the data is decrypted to obtain the decrypted text data, image data, and signal data. Sensitive information is matched against the text data using regular expressions, and the matched sensitive words are replaced with mask characters to obtain de-identified text data. Sensitive metadata is identified in the image data and the signal data, and the sensitive metadata is removed from the image data and the signal data respectively to obtain desensitized image data and desensitized signal data; The desensitized text data, the desensitized image data, and the desensitized signal data are identified as desensitized data of different modalities.

4. The method as described in claim 1, characterized in that, The desensitized data of different modalities includes: desensitized text data, desensitized image data, and desensitized signal data; Before the step of extracting features from the desensitized data of different modalities to determine the features of different modalities, the method further includes: The desensitized image data is subjected to rendering preprocessing to obtain preprocessed image data. The rendering preprocessing includes: window width and window level adjustment, noise reduction filtering, and multi-plane reconstruction. The desensitized signal data is subjected to signal correction processing to obtain processed signal data. The signal correction processing includes baseline drift correction and feature point localization. Accordingly, the step of extracting features from the desensitized data of different modalities to determine the different modal features includes: Feature extraction is performed on the desensitized text data, the preprocessed image data, and the processed signal data to obtain different modal features.

5. The method as described in claim 4, characterized in that, The different modal features include: semantic feature vectors, spatial feature vectors, and temporal feature vectors; The step of extracting features from the desensitized text data, the preprocessed image data, and the processed signal data to obtain different modal features includes: The desensitized text data is input into a preset entity extraction model to obtain the current text entity, and the current text entity is converted into a semantic feature vector, which is obtained by training with sample text data and sample text entities; The preprocessed image data is input into a preset image segmentation model to obtain a current lesion segmentation mask, and the current lesion segmentation mask is converted into a spatial feature vector. The preset image segmentation model is trained using sample image data and sample lesion segmentation masks. An abnormal signal segment is identified from the processed signal data using a dynamic time warping algorithm, and the abnormal signal segment is converted into a time-series feature vector.

6. The method as described in claim 5, characterized in that, The step of performing feature matching on the different modal features and constructing a dynamic association graph based on the matching results includes: Calculate the corresponding attention weights among the semantic feature vector, the spatial feature vector, and the temporal feature vector, respectively; A plurality of graph nodes are generated based on the semantic feature vector, the spatial feature vector and the temporal feature vector, the graph nodes including: text nodes, image nodes and signal nodes; Based on the corresponding attention weights, the connection relationships between the graph nodes are determined, and several connection edges are generated. By integrating all the graph nodes and all the connecting edges, a dynamic association graph is constructed.

7. The method as described in claim 6, characterized in that, The step of querying the associated data of the interaction based on the dynamic association graph and synchronously rendering the associated data to each of the modal views when receiving an interaction operation from a user on any of the modal views includes: When a user interacts with the current modal view, the first graph node corresponding to the interaction is determined, wherein the current modal view is a text view, an image view, or a signal view. The dynamic association graph queries other graph nodes that are connected to the first graph node, and the first graph node and the other graph nodes belong to different modal views; The first associated modal data corresponding to the first graph node is rendered to the current modal view, and the other associated modal data corresponding to the other graph nodes are rendered to other modal views.

8. The method as described in claim 1, characterized in that, After the step of synchronously rendering the associated data to each of the modal views, the method further includes: Obtain the updated modal view and receive feedback from the user on the updated modal view; The dynamic association graph is updated according to the feedback operation, and the association relationships between the modal views are redefined based on the updated dynamic association graph.

9. A multimodal fusion outpatient data processing system, characterized in that, The system includes: a front-end device and a back-end server. The back-end server is equipped with a heterogeneous medical platform. The front-end device includes: a data acquisition module, a front-end preprocessing module, a multimodal fusion module, and a rendering and interaction module. The data acquisition module is used to acquire patient outpatient data of different modalities from the heterogeneous medical platform; The front-end preprocessing module is used to desensitize the patient outpatient data of different modalities to obtain desensitized data of different modalities. The multimodal fusion module is used to extract features from the desensitized data of different modalities to determine different modal features. The multimodal fusion module is also used to perform feature matching on the different modal features and construct a dynamic association map based on the matching results; The rendering interaction module is used to generate modal views based on the desensitized data of different modalities and render them to the interactive page. The relationship between the modal views is defined by the dynamic association graph. The rendering interaction module is further configured to, upon receiving an interaction operation from a user on any of the modal views, query the associated data of the interaction operation based on the dynamic association graph, and synchronously render the associated data to each of the modal views.

10. A storage medium, characterized in that, The storage medium stores a multimodal fusion outpatient data processing program, which, when executed by a processor, implements the steps of the multimodal fusion outpatient data processing method as described in any one of claims 1 to 8.

Images