Multi-source medical data fusion method, system, medium and equipment

Abstract

The invention provides a multi-source medical data fusion method and system, a medium and equipment. The multi-source medical data fusion method comprises the steps of collecting multi-modal data from different data sources; performing feature extraction on the acquired multi-modal data by adopting a deep learning model to obtain feature data of different modal data; fusing the feature data of the different modal data to obtain fused data; and adjusting fusion weights of different modal data so as to carry out data quality control on the fusion data. According to the system, through a multi-source data fusion technology, the problem that medical data are scattered and inconsistent is solved, and the comprehensive utilization value of the data and the accuracy of analysis are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-source medical data fusion method, system, medium and device. Background Art

[0002] With the continuous development of society, computer technology and Internet technology have achieved unprecedented rapid development. Human society has entered the era of big data. The construction of medical industry informatization has also been accelerating, and the amount of various medical information data has shown an explosive growth trend.

[0003] Multi-source data fusion technology refers to the technology that uses relevant means to integrate all the information obtained through investigation and analysis, and uniformly evaluates the information to finally obtain unified information. The purpose of developing this technology is to integrate various different data information, absorb the characteristics of different data sources, and then extract unified information from them that is better and richer than single data.

[0004] In the medical industry, data is usually recorded in a mixed form in the form of multimodal data, namely text data, audio data, and video data. The premise of medical industry informatization must be the fusion and utilization of these multimodal data. However, traditional multimodal data fusion has problems with data fusion accuracy and poor consistency. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a multi-source medical data fusion method, system, medium and device.

[0006] The multi-source medical data fusion method provided by the present invention includes: collecting multimodal data from different data sources; using a deep learning model to extract features from the collected multimodal data to obtain feature data of different modal data; fusing the feature data of different modal data to obtain fused data; adjusting the fusion weights of different modal data to perform data quality control on the fused data.

[0007] It is not difficult to understand that the present invention can more comprehensively understand the patient's health status by fusing multimodal data from different data sources (such as imaging data, physiological signals, laboratory test results, etc.), thereby improving the accuracy of clinical diagnosis; further, the present invention uses a deep learning model for feature extraction, which can mine deep information from complex and large medical data sets, thereby enhancing the ability to understand and analyze the data; further, the present invention ensures the quality of the fused data through a data consistency check algorithm and anomaly detection mechanism, thereby reducing the risk of misdiagnosis due to data errors or anomalies.

[0008] Furthermore, the multimodal data includes image data, audio data and text data; feature extraction of the collected multimodal data includes: performing feature representation of the spatial hierarchical structure of the image data through a first model to generate first modal features; performing feature representation of time series data of the audio and text data through a second model to generate second modal features; and processing the contextual information of the text data through a self-attention mechanism of a third model to generate third modal features.

[0009] It is not difficult to understand that by using specialized models for different data types (such as images, audio, and text), it is possible to ensure that the most effective features are extracted from each modality, making the final feature representation more comprehensive. Combining the features of the three modalities together can produce a synergistic effect in multimodal learning tasks. For example, in video understanding and analysis, the complementarity of image, audio, and text information can enable the system to make more accurate decisions.

[0010] Furthermore, fusing feature data of different modal data includes: inputting first modal features, second modal features and third modal features into a pre-trained feature fusion model to obtain first modal fusion features, second modal fusion features and third modal fusion features; downsampling the first modal fusion features, second modal fusion features and third modal fusion features to obtain updated first modal fusion features, updated second modal fusion features and updated third modal fusion features; splicing the updated first modal fusion features, updated second modal fusion features and updated third modal fusion features to obtain fused data.

[0011] It is not difficult to understand that by fusing data features from different modalities, the advantages of each modality can be combined to obtain a more comprehensive information description, thereby improving the accuracy and reliability of the final analysis results. Processing data with a pre-trained feature fusion model can simplify the design difficulty of subsequent models, and due to the downsampling operation, it can also reduce the demand for computing resources and improve computing efficiency.

[0012] Furthermore, the pre-trained feature fusion model includes: concatenating the average value of each channel in the first modal feature, the average value of each channel in the second modal feature, and the average value of each channel in the third modal feature to obtain an intermediate fusion modality; obtaining a first weight, a second weight, and a third weight of the first modal feature, the second modal feature, and the third modal feature in the overall feature based on the intermediate fusion modality; multiplying the first weight by the first modal feature to obtain a first modal fusion feature; multiplying the second weight by the second modal feature to obtain a second modal fusion feature; and multiplying the third weight by the third modal feature to obtain a third modal fusion feature.

[0013] Furthermore, the method for adjusting the fusion weights of different modal data includes: judging whether the data quality of the fused data meets the preset requirements according to a preset confidence interval, and adjusting the first weight, the second weight and the third weight when the data quality of the fused data does not meet the preset requirements.

[0014] Furthermore, the method for judging whether the data quality of fused data meets preset requirements according to a preset confidence interval includes: predicting the first modal feature, the second modal feature and the third modal feature respectively to generate a first decision result, a second decision result and a third decision result; obtaining deviation values ​​of the first decision result, the second decision result and the third decision result respectively according to the preset confidence interval; wherein, adjusting the corresponding first weight, second weight and third weight according to the deviation value.

[0015] It is not difficult to understand that by monitoring the deviations of the prediction results of each modality data (i.e., the first, second, and third modality features) and adjusting their weights in the final fusion data based on these deviations, it is possible to ensure that those modalities that provide more accurate information occupy a more important position in the decision-making process. When the data quality of a certain modality decreases or becomes abnormal, the influence of this modality can be weakened by reducing its weight, while the data of other modalities will increase their influence accordingly to compensate for this deficiency.

[0016] Furthermore, when the data quality of the fused data meets the preset requirements, the fused data is sent to the user interface for display.

[0017] The multi-source medical data fusion system provided by the present invention includes the following modules:

[0018] Data collection module: collects multimodal data from different data sources;

[0019] Multimodal learning module: uses a deep learning model to extract features from the collected multimodal data to obtain feature data of different modal data;

[0020] Data fusion module: fuses the feature data of different modal data to obtain fused data;

[0021] Data quality control module: adjusts the fusion weights of different modal data to perform data quality control on the fused data.

[0022] According to the computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the multi-source medical data fusion method are implemented.

[0023] The electronic device provided according to the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the steps of the multi-source medical data fusion method are implemented when the computer program is executed by the processor.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention uses multi-source data fusion technology to integrate and coordinate medical data from different sources, uses multimodal learning technology and data fusion algorithms to generate comprehensive data representation, and ensures data accuracy and consistency through data quality control, thereby solving the problems of medical data dispersion and inconsistency, improving the comprehensive utilization value of data and the accuracy of analysis, and significantly improving the efficiency and quality of data fusion through multimodal learning and adaptive data fusion algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 It is a flow chart of the multi-source medical data fusion method;

[0028] Figure 2 A flowchart for fusing feature data of different modal data;

[0029] Figure 3 The flowchart of the method for judging data quality is as follows;

[0030] Figure 4 It is a schematic diagram of the data fusion algorithm module structure;

[0031] Figure 5 This is a diagram of the multi-source medical data fusion framework. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0033] The present invention provides a multi-source medical data fusion method, comprising: collecting multimodal data from different data sources; using a deep learning model to perform feature extraction on the collected multimodal data to obtain feature data of different modal data; fusing the feature data of different modal data to obtain fused data; performing data quality control on the fused data and adjusting the fusion weight of the feature data of different modal data.

[0034] It is not difficult to understand that the present invention can more comprehensively understand the patient's health status by fusing multimodal data from different data sources (such as imaging data, physiological signals, laboratory test results, etc.), thereby improving the accuracy of clinical diagnosis; further, the present invention uses a deep learning model for feature extraction, which can mine deep information from complex and large medical data sets, thereby enhancing the ability to understand and analyze the data; further, the present invention adjusts the fusion weight of feature data of different modal data to ensure the quality of the fused data and reduce the risk of misdiagnosis due to data errors or anomalies.

[0035] like Figure 1 The present invention provides a multi-source medical data fusion method, comprising:

[0036] Step S1: Collect multimodal data from different data sources.

[0037] Specifically, the data source can be medical equipment, databases, and electronic health records in different medical institutions and community groups. Different data sources collect data by configuring different interfaces, using medical data standards such as DICOM, HL7, and FHIR. A data collection node and a data transmission node can be configured in each medical institution. The data collection node collects data through different interfaces and then uploads the collected medical data through the data transmission node. Multimodal data includes image data, audio data, and text data.

[0038] In some embodiments, the present invention can be used to provide a summary of the health conditions of residents in the service area and alerts to people at risk of chronic diseases. In response to user operations, the resident information in the service area is obtained. The resident information can be resident health data summarized by the community neighborhood committee, resident medical treatment data recorded by local hospitals, health data uploaded by individual residents, medication data, living habit data, etc.

[0039] Step S2: Use a deep learning model to extract features from the collected multimodal data to obtain feature data of different modal data.

[0040] Specifically, multimodal data includes image data, audio data and text data. Feature extraction of the collected multimodal data includes: using a first model to represent the spatial hierarchical structure of image data to generate first modal features; using a second model to represent the time series data of audio and text data to generate second modal features; and using a third model to process the contextual information of text data through a self-attention mechanism to generate third modal features.

[0041] In some embodiments, spatial hierarchical feature representation of image data through a convolutional neural network CNN includes: dividing the image data into three channels of RGB to obtain feature maps of the three channels, inputting multiple convolution layers into the three channels respectively to obtain local features of several images, inputting the local features obtained by convolution into an attention layer to obtain global features of the image, and flattening the global features of the image to obtain image modal features.

[0042] The feature representation of time series data of audio data and text data through recurrent neural network RNN ​​and long short-term memory network LSTM includes: first, the audio signal needs to be converted into a form suitable for input to the neural network, such as a spectrogram or Mel-frequency cepstral coefficients MFCCs, and the text needs to be converted into a numerical vector through word embedding encoding, such as Word2Vec or GloVe; then, these sequence data are trained using an RNN model with LSTM units to capture the temporal dependency in the data, thereby generating audio / text modal features.

[0043] The context information of text data is processed through the self-attention mechanism of the Transformer model, including: inputting text data into the Transformer model to obtain global features and local context features of the text data, and fusing the global features and local context features to further enhance the model's understanding of the sequence data and the context of the text data, thereby generating text modal features after semantic understanding. The purpose of this is that although the text data has been converted into text modal features in the previous step, this direct way of converting the text will lead to different conversion results between synonyms and synonyms, and inaccurate conversion results for polysemous words. Therefore, the text data is processed again here so that the obtained text modal features after semantic understanding can accurately express the meaning of the text data.

[0044] Furthermore, in actual operation, it is necessary to tune the feature extraction process of each modality, such as selecting appropriate network architecture parameters, loss function, and optimizer, while ensuring that there is enough labeled data for training so that the model can learn effective feature representation.

[0045] In order to make the deep learning model more efficient and accurate in feature extraction of the collected multimodal data, before step S2, it can also include: cleaning, formatting and normalizing the collected data. Further, cleaning, formatting and normalizing the collected data include: first, the goal of data cleaning is to identify and correct or delete erroneous, incomplete, inaccurate or irrelevant data records, which usually involves checking the consistency of the data, such as removing duplicates, filling missing values, correcting format errors, etc.; for text data, it may be necessary to remove special characters and stop words, and perform stem extraction or word form restoration; for image data, it may be necessary to crop, rotate or adjust the brightness contrast to standardize the image; and audio data may need to remove background noise or silent paragraphs; secondly, formatting is to make the data conform to specific standards or patterns for unified processing, for example, all texts can be converted to lowercase, the date format can be unified, and images can be converted to the same resolution and color mode; finally, normalization is to eliminate the influence of data dimensions so that data of different scales can be compared and calculated under the same standard. Common methods include min-max normalization (scaling the data to between 0 and 1), Z-score standardization (making the data have zero mean and unit variance), etc. This process usually involves calculating the maximum, minimum, or average value of each feature and adjusting the data accordingly. In practice, these steps can be automated by writing scripts, using the Pandas library in programming languages ​​such as Python to process tabular data, the OpenCV library to process image data, and the Librosa library to process audio data. In addition, specialized data cleaning tools or ETL (extraction, transformation, loading) tools can be used to assist in processing large-scale data sets. Through these steps, the quality of the data can be significantly improved, ensuring that subsequent data analysis or modeling work is more efficient and accurate.

[0046] Step S3: Fusing the feature data of different modal data to obtain fused data.

[0047] Specifically, Figure 2 , the fusion of feature data of different modal data includes:

[0048] Step S301, inputting image modal features, audio / text modal features and text modal features after semantic understanding into a pre-trained feature fusion model to obtain intermediate fusion modal features. The feature fusion model performs feature fusion according to a preset first weight corresponding to the image modal features, a second weight corresponding to the audio / text modal features and a third weight corresponding to the text modal features after semantic understanding. The preset first weight, the preset second weight and the preset third weight can be set based on experience.

[0049] Step S302: Based on the intermediate fusion modality, the first weight, second weight, and third weight of the image modality feature, the audio / text modality feature, and the text modality feature after semantic understanding in the intermediate fusion modality feature are obtained. The first weight, second weight, and third weight referred to here correspond to the first weight, second weight, and third weight used in the feature fusion in step S301.

[0050] Step S303: multiply the first weight by the image modal feature to obtain the first modal fusion feature; multiply the second weight by the audio / text modal feature to obtain the second modal fusion feature; multiply the third weight by the text modal feature after semantic understanding to obtain the third modal fusion feature.

[0051] Step S304, downsample the first modal fusion feature to obtain an updated image modal feature; downsample the second modal fusion feature to obtain an updated audio / text modal feature; downsample the third modal fusion feature to obtain an updated text modal feature after semantic understanding. It should be noted that the fusion feature usually contains a variety of feature information from different modalities or different levels. These feature information may have a higher dimension or resolution after fusion, and direct processing may bring a large computational burden. Therefore, it is necessary to downsample the fusion feature to reduce the feature dimension or resolution, which can reduce the amount of calculation in subsequent processing steps and improve the processing speed.

[0052] Step S305: splice the updated image modality features, audio / text modality features, and text modality features after semantic understanding to obtain fused data.

[0053] Step S4: Adjust the fusion weights of different modal data to perform data quality control on the fused data.

[0054] Specifically, the method for adjusting the fusion weights of different modal data includes: judging whether the data quality of the fused data meets the preset requirements according to the preset confidence interval, and adjusting the first weight, the second weight and the third weight when the data quality of the fused data does not meet the preset requirements. Figure 3 , the methods for judging data quality include:

[0055] Step S401, predict the image modal features, audio / text modal features and text modal features after semantic understanding respectively, and generate a first decision result, a second decision result and a third decision result. The first decision result can be to increase / decrease the first weight, the second decision result can be to increase / decrease the second weight, and the third decision result can be to increase / decrease the third weight.

[0056] Step S402: Calculate the corresponding confidence intervals according to the first decision result, the second decision result, and the third decision result to obtain the deviations of the three decision results. The confidence interval provides the uncertainty range of the modal data, which helps to evaluate its stability and reliability. Generally, modal data with narrower confidence intervals means that the uncertainty of its estimated value is smaller, so it can be given a higher weight in the fusion process.

[0057] Step S403: adjusting the corresponding first weight, second weight and third weight according to the deviation to achieve more accurate and reliable multimodal data fusion.

[0058] Furthermore, after the weight is adjusted in step S403, a new confidence interval is formed. By adjusting the weight so that the narrower the confidence interval is, the higher the quality of the fused data is. When the data quality of the fused data meets the preset requirements, the fused data is sent to the user interface for display.

[0059] Specifically, in some embodiments, a comprehensive method integrating front-end technology and back-end architecture can be used to provide a user interface and interactive functions, and the method includes: first, in terms of front-end design, a modern web framework such as React, Vue.js or Angular is selected, which provides the ability to build dynamic user interfaces and can easily handle complex user interactions. For example, through React, we can use its virtual DOM technology to optimize UI updates and improve application performance. At the same time, state management tools such as Redux or Vuex can be used to maintain the state of the application, so that the state update logic is clear and easy to manage. For the design of the user interface, attention should be paid to the best practices of user experience (UX) and user interface (UI), including responsive layout to adapt to different devices, and intuitive operation processes. Using CSS preprocessors such as Sass or Less can help us write more modular and maintainable style sheets, while UI libraries such as Bootstrap or Material-UI can quickly build professional appearance designs. In terms of interactive functions, the front end needs to communicate with the back end. This is usually implemented through RESTful API or GraphQL interfaces, which allow data to be requested from the front end or sent to the server. WebSocket technology can be used to achieve real-time two-way communication, which is very useful for application scenarios that require instant feedback, such as chat applications or collaborative editing tools. The backend can choose platforms such as Node.js, Django, or Ruby on Rails to develop API services. These frameworks support functions such as database operations, user authentication and authorization, ensuring the security of the system. The choice of database is also very important. Relational databases such as MySQL or PostgreSQL are suitable for complex queries, while non-relational databases such as MongoDB are suitable for large-scale data storage and high-concurrency access scenarios. Finally, the deployment plan is also key. Containerization technologies such as Docker combined with Kubernetes cluster management can realize automated deployment, expansion and operation and maintenance. In addition, security and performance optimization measures need to be considered, such as HTTPS encryption, CDN acceleration, and reasonable setting of cache strategies.

[0060] The present invention also proposes a multi-source medical data fusion system for implementing the above-mentioned multi-source medical data fusion method. The system can integrate and coordinate medical data from different sources, use multimodal learning technology and data fusion algorithm to generate comprehensive data representation, and ensure the accuracy and consistency of the data through data quality control.

[0061] like Figure 5 , the system includes the following modules:

[0062] Data collection module 101: collects medical data from different data sources, which can be medical equipment, databases and electronic health records in different medical institutions and community groups. Different data sources collect data by configuring different interfaces, using medical data standards such as DICOM, HL7, and FHIR. A data collection node and a data transmission node can be configured in each medical institution. After the data collection node collects data through different interfaces, the collected medical data is transmitted to the data preprocessing module and the multimodal learning module through the data transmission node.

[0063] For example, in response to the user's operation, the resident information in a specific area is obtained, and the resident information can be the resident health data summarized by the community neighborhood committee, the resident medical data recorded by the local hospital, the health data uploaded by the resident, the medication data, the living habit data, etc. Therefore, the present invention can be used to provide a summary of the health status of residents in the service area and a warning for people at risk of chronic diseases.

[0064] Data preprocessing module 102: preprocessing the residents' data, including: cleaning, formatting and normalizing the collected data according to preset conditions; developing an automated data cleaning algorithm to remove noise and outliers and achieve data standardization; transmitting the preprocessed data to the multimodal learning module.

[0065] Multimodal learning module 103: Use multimodal learning technology to process different types of data, such as images, text and numerical data; use deep learning models, such as CNN, RNN and Transformer, to extract features from different modal data; and transmit the extracted feature data to the data fusion algorithm module.

[0066] Multimodal data includes images, videos, text, and numerical data. The image data of this application includes: CT images, B-ultrasound images, medical diagnosis images, prescription records, etc. Text data includes: question and answer data, personal information data, doctor's diagnosis and treatment records, etc. Numerical data includes: medication dosage data.

[0067] Data fusion algorithm module 104: using data fusion algorithms, such as Kalman filtering or Bayesian methods, to integrate feature data from different sources and types; designing adaptive data fusion strategies to dynamically adjust fusion weights according to data quality and relevance; transmitting the fused comprehensive data representation to the data quality control module.

[0068] like Figure 4, a convolutional neural network (CNN) is used to represent the spatial hierarchical structure of image and video data, and a first modal feature 201 is generated. A recurrent neural network (RNN) and a long short-term memory (LSTM) are used to represent the time series data of audio and text, and a second modal feature 202 is generated. The Transformer model efficiently processes long sequences and context information through a self-attention mechanism to generate a third modal feature 203. The first modal feature 201, the second modal feature 202, and the third modal feature 203 are input into a pre-trained feature fusion model 204, and after feature fusion, a fused feature is obtained.

[0069] Further, the average value of each channel in the first modal feature, the average value of each channel in the second modal feature and the average value of each channel in the third modal feature are concatenated to obtain an intermediate fusion modality; based on the intermediate fusion modality, the first weight, second weight and third weight of the first modal feature, the second modal feature and the third modal feature in the overall feature are obtained respectively; the first weight is multiplied by the first modal feature to obtain the first modal fusion feature; the second weight is multiplied by the second modal feature to obtain the second modal fusion feature; the third weight is multiplied by the third modal feature to obtain the third modal fusion feature; the first modal fusion feature is downsampled to obtain an updated first modal feature; the second modal fusion feature is downsampled to obtain an updated second modal feature; the third modal fusion feature is downsampled to obtain an updated third modal feature; the updated first modal feature, the updated second modal feature and the updated second modal feature are concatenated to obtain fusion data 205.

[0070] Data quality control module 105: detects and processes inconsistencies and errors in the data; implements data consistency check algorithms and anomaly detection mechanisms, such as methods based on statistical thresholds or machine learning; feeds quality control results back to the data fusion algorithm module, and provides cleaned data to the user interface module.

[0071] Furthermore, the data quality control module is used to adjust the first weight, the second weight, and the third weight, including: judging whether the data quality is normal according to the confidence interval, if the data quality is poor, adjusting the first weight, the second weight, and the third weight, and feeding the adjusted first weight, second weight, and third weight back to the previous step; predicting the first modal feature, the second modal feature, and the third modal feature respectively to generate the first decision result, the second decision result, and the third decision result; judging the deviation of the first decision result, the second decision result, and the third decision result respectively according to the confidence interval, and adjusting the first weight, the second weight, and the third weight according to the deviation value; if the data quality of the fused data is good, the fused data is integrated and sent to the user interface module for display.

[0072] User interface module 106: provides a user interface for medical professionals to access and analyze the fused data; implements data visualization tools to support data query, comparison and analysis functions; receives user query requests and displays fused data and analysis results.

[0073] It also includes a system integration and coordination module: ensuring that the system modules work together, conducting system testing and performance evaluation; implementing interface standardization and data synchronization mechanisms between modules, conducting integration testing and stress testing; and feeding back system integration and test results to each module to guide system optimization and upgrades.

[0074] Specifically, in some embodiments, the working mechanism of the system integration and coordination module includes: first, establish a comprehensive system testing framework, which should cover unit testing, integration testing and system-level performance evaluation. In the early stage of development, the standard interface protocol for communication between modules should be defined, which usually includes data format, transmission protocol and error handling mechanism, etc., to ensure data synchronization and consistency between different modules. After the integration test, stress testing should be carried out to evaluate the performance boundary of the system, identify potential bottlenecks and optimize them, such as improving the stability and response speed of the system through load balancing, caching strategy or database optimization. The data collected during the test and the problems encountered should be fed back to the corresponding development team in a timely manner so that they can adjust and optimize the modules they are responsible for in a targeted manner. In addition, the practice of continuous integration / continuous deployment (CI / CD) is adopted, so that the test script can be automatically run after each code submission, problems can be quickly discovered and rapid iterations can be promoted.

[0075] It also includes a deployment and maintenance module: responsible for system deployment, monitoring and maintenance to ensure stable system operation; integrated monitoring system to track system performance indicators in real time and implement automated deployment and rollback mechanisms; data transmission: collect system operation data and maintenance logs for continuous system improvement.

[0076] Specifically, in some embodiments, the working mechanism of the deployment and maintenance module includes: first, in the deployment stage, containerization technology such as Docker is used to encapsulate applications and their dependencies to ensure environmental consistency, and Kubernetes (K8s) is used for cluster management to achieve automatic expansion and contraction of services and load balancing. With the help of tools such as Jenkins or GitLab CI / CD, a continuous integration / continuous deployment (CI / CD) pipeline is built to automatically complete the code building, testing and deployment processes, and an automated rollback strategy is configured to deal with deployment failures. Secondly, in terms of monitoring, open source solutions such as Prometheus and Grafana are integrated to establish a comprehensive monitoring system to monitor key performance indicators such as CPU, memory, and disk I / O, and set a reasonable threshold alarm mechanism to respond to potential failures in a timely manner. For log management, the ELK (Elasticsearch, Logstash, Kibana) stack can be used to collect, parse and visualize various types of log information, which not only helps to quickly locate problems, but also provides a basis for subsequent data analysis. Finally, by regularly analyzing system operation data and logs, identifying bottlenecks, optimizing system architecture and code, and continuously improving system stability and efficiency. Throughout the process, attention should also be paid to the implementation of security protection measures, such as encrypted storage of sensitive data, regular security scans, and allocation of access control in accordance with the principle of least privilege, so as to build an operation and maintenance environment that is both efficient and secure.

[0077] It should be understood that the embodiments of the present invention can be implemented in hardware, software or a combination thereof. Software can be stored in a volatile or non-volatile storage device (regardless of whether the data is deletable or rewritable) like a ROM, in a memory like a RAM, a memory chip, a device or an integrated circuit, or in a storage medium on which data can be optically or magnetically recorded and from which data can be read by a machine (e.g., a computer), such as a compact disc (CD), a digital versatile disc (DVD), a disk, or a tape. Moreover, the embodiments of the present invention can be implemented in a computer or portable terminal having a controller and a memory, and the memory is an example of a non-temporary computer-readable storage medium suitable for storing one or more programs including commands for implementing the embodiments of the present invention. Therefore, the present invention includes a program having a code for implementing the device or method defined by the claims, and a machine-readable non-temporary storage medium storing the program. The program can be transferred electrically by a medium such as a communication signal transmitted via a wired or wireless connection, and the program and its equivalent are included in the present invention.

[0078] The device may receive a program from a program providing device connected via a cable or wirelessly and store the program. The program providing device may include a program containing commands for implementing the embodiments of the present invention, a memory for storing information required by the embodiments of the present invention, a communication module for communicating with the mobile device via a cable or wirelessly, and a controller for transmitting the program to the device automatically or upon request of the device.

[0079] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-source medical data fusion method, characterized in that: include: Collect multimodal data from different data sources; Using deep learning models, feature extraction is performed on the collected multimodal data to obtain feature data of different modal data; Fusing feature data of different modal data to obtain fused data; Adjust the fusion weights of different modal data to perform data quality control on the fused data.

2. The multi-source medical data fusion method according to claim 1, characterized in that: Multimodal data includes image data, audio data, and text data; Feature extraction of collected multimodal data includes: Performing spatial hierarchical feature representation on the image data through a first model to generate a first modality feature; Performing feature representation of time series data on audio and text data through a second model to generate second modal features; The contextual information of the text data is processed through the self-attention mechanism of the third model to generate the third modality features.

3. The multi-source medical data fusion method according to claim 2, characterized in that: The fusion of feature data of different modal data includes: Inputting the first modality feature, the second modality feature, and the third modality feature into a pre-trained feature fusion model to obtain the first modality fusion feature, the second modality fusion feature, and the third modality fusion feature; Down-sampling the first modality fusion feature, the second modality fusion feature, and the third modality fusion feature to obtain an updated first modality fusion feature, an updated second modality fusion feature, and an updated third modality fusion feature; The updated first modality fusion feature, the updated second modality fusion feature, and the updated third modality fusion feature are concatenated to obtain fused data.

4. The multi-source medical data fusion method according to claim 3, characterized in that: The pre-trained feature fusion model includes: The average value of each channel in the first modality feature, the average value of each channel in the second modality feature, and the average value of each channel in the third modality feature are concatenated to obtain an intermediate fusion modality; Based on the intermediate fusion mode, a first weight, a second weight, and a third weight of the first modality feature, the second modality feature, and the third modality feature in the overall feature are obtained respectively; The first weight is multiplied by the first modal feature to obtain the first modal fusion feature; the second weight is multiplied by the second modal feature to obtain the second modal fusion feature; the third weight is multiplied by the third modal feature to obtain the third modal fusion feature.

5. The multi-source medical data fusion method according to claim 4, characterized in that: Methods for adjusting the fusion weights of different modal data include: Whether the data quality of the fused data meets the preset requirements is judged according to the preset confidence interval, and when the data quality of the fused data does not meet the preset requirements, the first weight, the second weight and the third weight are adjusted.

6. The multi-source medical data fusion method according to claim 5, characterized in that: Methods for judging whether the data quality of fused data meets preset requirements according to preset confidence intervals include: Predicting the first modal feature, the second modal feature, and the third modal feature respectively to generate a first decision result, a second decision result, and a third decision result; Obtaining deviation values ​​of the first decision result, the second decision result, and the third decision result respectively according to a preset confidence interval; Wherein, the corresponding first weight, second weight and third weight are adjusted according to the deviation value.

7. The multi-source medical data fusion method according to claim 5, characterized in that: When the data quality of the fused data meets the preset requirements, the fused data is sent to the user interface for display.

8. A multi-source medical data fusion system, characterized in that: The multi-source medical data fusion method according to any one of claims 1 to 7 comprises: Data collection module: collects multimodal data from different data sources; Multimodal learning module: uses a deep learning model to extract features from the collected multimodal data to obtain feature data of different modal data; Data fusion module: fuses the feature data of different modal data to obtain fused data; Data quality control module: adjusts the fusion weights of different modal data to perform data quality control on the fused data.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-source medical data fusion method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the multi-source medical data fusion method according to any one of claims 1 to 7 are implemented.

Images