Intelligent view synchronization and communication system and method for multi-modal medical image comparison

Abstract

The invention discloses an intelligent view synchronization and communication system and method for multi-modal medical image comparison. According to the system, medical images of different modalities are accurately aligned through an image registration algorithm, and synchronous display and interaction at a doctor end and a patient end are realized by adopting a view synchronization mechanism. The collaboration server sends the image data to each terminal, and the local rendering image ensures high definition; and when the doctor terminal zooms, translates or marks the image, the system generates a synchronization instruction and transmits the synchronization instruction to the patient terminal, so that the patient terminal updates a corresponding view in real time. Particularly, spatial mapping parameters obtained through registration are utilized to ensure that anatomical positions among different modal image views correspond, a focus positioned by a doctor in one modal can automatically display the corresponding position on an image in the other modal in a highlight mode, and remote comparison and analysis of both parties are facilitated. According to the invention, the efficiency and accuracy of multi-modal medical image comprehensive diagnosis in remote consultation are obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and telemedicine, in particular to an intelligent view synchronization and communication system and method for multi-modal medical image comparison. BACKGROUND

[0002] Telemedicine utilizes communication technology, medical imaging technology and computer multimedia technology to provide diagnostic capabilities of large hospitals through a network to remote or weak medical resource areas. For example, based on the Dicom3.0 standard transmission of medical image files, CT, MRI and other image data obtained by hospitals at different locations can be transmitted to a central hospital, and the results are fed back after being diagnosed by experts, realizing the sharing of medical images in multiple places. Through the remote image diagnosis system, the lower-level hospital transmits the patient image to the expert of the higher-level hospital for reading and interaction, improving the diagnostic accuracy and efficiency.

[0003] There are mainly two schemes for existing medical image collaborative viewing methods: one is to use remote desktop sharing or application program sharing, and each end transmits screenshots for image exchange. This way transmits interface image data, which can cause distortion of medical images and cannot guarantee the authenticity and accuracy of DICOM images. The second is a collaborative method based on a cloud server, for example, the client interacts with the cloud through WebSocket, performs all operations on the server, and then synchronizes the result image to each collaborative end. Under this scheme, the image display is almost distortion-free, but when a new view needs to be generated, all user ends need to wait for the server to re-render the image, consuming a lot of bandwidth and computing power, and reducing the collaboration speed. The above schemes do not fully consider the comparison requirements of multi-modal medical images: due to the differences in imaging principles, different modalities (such as MRI and CT, ultrasound, etc.) have differences in spatial scale and anatomical structure display. If there is no effective registration and synchronization mechanism, doctors will have difficulty in accurately positioning the same lesion between different modalities for comparison and analysis during remote consultation, affecting the diagnosis efficiency and accuracy. SUMMARY

[0004] Technical purpose: In view of the deficiencies of the prior art, the present application discloses an intelligent view synchronization and communication system and method for multi-modal medical image comparison, which realizes synchronous display and interaction of medical images of multiple modalities on the remote doctor end and the patient end by introducing image registration and view synchronization algorithms, and realizes off-site expert collaborative reading and real-time discussion by combining an efficient network communication mechanism.

[0005] Technical scheme: In order to achieve the above technical purpose, the present application adopts the following technical scheme:

[0006] An intelligent view synchronization and communication method for multi-modal medical image comparison, comprising the following steps:

[0007] acquire first modality medical images and second modality medical images to be compared, and pre-process the multi-modality medical images to unify image format and spatial resolution;

[0008] perform image registration on the first modality medical images and the second modality medical images to obtain spatial transformation parameters mapping the first modality image coordinates to the second modality image coordinates;

[0009] establish network communication connection between the remote consultation participating terminals, send the first modality and second modality medical image data to the user terminals of the consultation, and synchronize the spatial transformation parameters to the user terminals;

[0010] display the first modality medical images and the second modality medical images on the first user terminal and the second user terminal respectively, listen to the image interaction operations on the first user terminal, and convert the interaction operations into synchronization instructions in real time and send them to the second user terminal through the network;

[0011] the second user terminal receives and executes the synchronization instructions, updates the corresponding view of the locally displayed medical images, so that the image view displayed by the second user terminal is synchronized with the interaction operations of the first user terminal; if the interaction operations involve positioning a target position on the first modality image, the target position is mapped to the corresponding position on the second modality image using the spatial transformation parameters for synchronized display;

[0012] when any user terminal receives an image annotation instruction from another user terminal, add a corresponding mark to the corresponding position of the local image for display, and record and save the mark information for the consultation report;

[0013] end the remote collaborative consultation as needed, release the communication connection resources after the consultation is completed, and save the marks and interaction records generated during the consultation.

[0014] Preferably, the image registration includes first performing initial registration according to the metadata or anatomical landmark points of the medical images, and then iteratively solving the accurate alignment transformation of the multi-modality medical images using an optimization algorithm; the optimization algorithm uses an evaluation function combining statistical intensity similarity and anatomical structure similarity to guide the registration, wherein the evaluation function includes a weighted combination of mutual information indicators and edge overlap indicators of the images, to improve the robustness and accuracy of multi-modality image registration.

[0015] Preferably, the synchronization instructions include image identification, view parameters and spatial transformation parameters involved in the interaction operations; the first user terminal and the second user terminal each locally save complete medical image data, and the execution of the synchronization instructions is realized by re-rendering the image view locally, without the need to repeatedly transmit large-capacity image pixel data.

[0016] Preferably, the first user terminal and the second user terminal both support multi-window multi-image synchronous linkage display, and the spatial transformation parameter is used to ensure that the view positions of the two modal medical images remain in an anatomical correspondence relationship when the first user terminal switches the view linkage mode between the first modal medical image and the second modal medical image; when the user scrolls and browses in the medical image window of one modal, the medical image window of the other modal automatically jumps to the image at the corresponding anatomical position, realizing synchronous comparison browsing across modalities.

[0017] Preferably, a master control permission switching mechanism is included, and initially, the user terminal of the party that initiates the consultation serves as the master terminal, and synchronizes its interactive operation to all other participating terminals; when another participating terminal requests to obtain the master control permission and obtains consent, the switching is performed by the requesting terminal as the new master terminal, and the subsequent interactive operation is synchronized to the remaining terminals, and the original master terminal becomes a passive synchronization state; the switching of the master control permission is coordinated by the collaborative server, ensuring that the image view states of the terminals are continuous and consistent and the synchronization is uninterrupted.

[0018] An intelligent view synchronization and communication system for multi-modal medical image comparison, for realizing an intelligent view synchronization and communication method for multi-modal medical image comparison as described above, comprising at least two user terminal devices and a collaborative server in communication connection with the user terminal devices,

[0019] The user terminal device comprises an image display and interaction module for loading and displaying medical images for user browsing and interaction; a view synchronization client module for capturing local user image interaction operations and generating synchronization instructions, and receiving remote synchronization instructions and executing locally to update the image view; The collaborative server comprises an image data storage unit for storing multi-modal medical image data of a patient; a registration calculation unit for calculating spatial transformation parameters between multi-modal medical images; a communication coordination unit for establishing and managing a collaborative session between multiple user terminal devices, and transmitting medical image data and view synchronization instructions between user terminal devices and the server through a network; The spatial transformation parameters calculated by the registration calculation unit are used to guide the view synchronization client module to perform view synchronization display between corresponding positions of different modal medical images, and the communication coordination unit is configured to transmit initial medical image data through an independent first data channel and transmit synchronization instructions through an independent second data channel, so as to reduce data conflicts and improve synchronization real-time performance.

[0020] Preferably, the registration calculation unit is configured to automatically align the multi-modal medical images by using a registration algorithm based on mutual information and edge feature fusion, and send the obtained spatial transformation matrix to each user terminal device; the view synchronization client module converts a coordinate point on one modal medical image into a corresponding coordinate point on another modal medical image according to the spatial transformation matrix, so as to synchronously display the anatomically corresponding markers or cursor positions in the different modal medical image views of each user terminal.

[0021] Preferably, the communication coordination unit further comprises a permission management component for switching the master permission of view synchronization among the plurality of user terminal devices, so that the interactive operation of the user terminal device currently having the master permission is synchronously broadcast to other user terminal devices, and the consistency of the medical image display state of each user terminal device is maintained when the master permission is switched.

[0022] Preferably, the user terminal device further comprises an annotation marker module for accepting the graphical or text markers added to the medical images by the user; the view synchronization client module sends the markers added in one user terminal to other user terminal devices through the communication coordination unit, and maps the markers to the correct positions of the corresponding modal medical images of other user terminal devices for display by using the spatial transformation parameters, so as to realize the real-time sharing of the marker information; the annotation marker module is further configured to save all the markers for viewing after consultation.

[0023] Preferably, the communication coordination unit adopts an encryption algorithm to securely protect the transmitted medical image data and synchronization instruction data, and performs identity authentication on the connected user terminal devices, so as to ensure the patient privacy and data security; the system has a reconnection and continuous transmission function after network interruption, and can automatically resend the key synchronization instructions not executed during the interruption after the connection is restored, so as to guarantee the integrity of the remote collaboration process.

[0024] Beneficial effects: The intelligent view synchronization and communication system and method for multi-modal medical image comparison provided by the application have the following beneficial effects:

[0025] Firstly, the system can accurately align and synchronously display medical images of different modalities in a remote scenario. Doctors and patients can simultaneously observe the contrast of the same anatomical part under different imaging, even if they are separated by thousands of miles. This facilitates the discovery of complementary information of each modality and improves the comprehensiveness of diagnosis. Secondly, the application adopts a local rendering combined with instruction synchronization architecture to minimize network transmission load while ensuring image quality without distortion. The low interaction delay improves the smoothness of collaborative reading. At the same time, the intelligent view synchronization and marker sharing function of the application enables multiple consultants to exchange views and jointly label lesions in real time, greatly improving the efficiency and accuracy of remote consultation. Especially for complex diseases that require comprehensive interpretation of MRI, CT, ultrasound, etc., the system provides a powerful collaborative diagnosis tool with significant clinical application value. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description.

[0027] Figure 1 It is a schematic diagram of the system architecture of the present application.

[0028] Figure 2 It is a schematic diagram of the method flow of the present application.

[0029] Figure 3 It is an interface diagram of the multi-modal image synchronous display in the specific embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application will be more clearly and completely described by means of a preferred embodiment and in conjunction with the accompanying drawings.

[0031] As shown in the drawings, an intelligent view synchronization and communication method for multi-modal medical image contrast includes the following steps: Figure 2

[0032] S1, obtaining the first modality medical image and the second modality medical image to be compared, and pre-processing the multi-modal medical image to unify the image format and spatial resolution.

[0033] The multi-modal medical image can be MRI image and CT image. The pre-processing operation is, for example, denoising, gray correction and format conversion, to extract the anatomical structure information in the image.

[0034] S2, performing image registration on the first modality medical image and the second modality medical image to obtain spatial transformation parameters for mapping the first modality image coordinates to the second modality image coordinates.

[0035] ​The improved multi-modal image registration algorithm is used to align the first modality medical image and the second modality medical image. The registration process includes two stages of initial geometric alignment and fine registration: the initial alignment can calculate the initial transformation guess based on the medical image metadata or the anatomical landmark points (such as the center of the patient's anatomical structure); the fine registration is based on an optimization algorithm to solve the accurate spatial transformation parameters. In order to improve the accuracy of multi-modal registration, the application proposes a matching evaluation function which integrates statistical information and edge structure. In the registration optimization, the similarity evaluation function of two medical images I A (such as CT) and I B (such as MRI) under the spatial transformation parameter T is defined as:

[0036]

[0037] Wherein, wherein, and respectively represent the information entropy of the medical images I A and I B , represent the joint entropy of the two medical images; represents the image obtained by transforming the medical image I B through the spatial transformation parameter T; is the edge indicator function of pixel x in the medical image I A (for example, based on gradient amplitude thresholding, E(x)=1 indicates that the pixel x belongs to the anatomical edge, otherwise E(x)=0), is the edge indicator function of the corresponding position of pixel x after being mapped to I B through the spatial transformation parameter T; represents the spatial pixel set defined by the medical image; , are respectively the mutual information item weight coefficient and the edge overlap item weight coefficient, which are used to balance the contribution of statistical similarity and structural similarity to registration. The first item of the evaluation function is the normalized mutual information, which measures the consistency of image gray distribution and can cope with the gray difference of multi-modal images; the second item is the edge overlap rate, which measures the matching degree of the anatomical profiles of the two images and emphasizes the matching of structural features. By maximizing S(T), the optimal spatial transformation parameter is solved. The optimal spatial transformation parameter can be represented as a rigid or affine transformation matrix, for example, in the form of a 4x4 matrix in homogeneous coordinates in a three-dimensional space:

[0038]

[0039] Wherein, wherein r ij (i,j=1,2,3) represents the rotation and scaling components, t x , ty , t z denotes the translation component. This matrix defines the spatial mapping relationship from the first modality medical image coordinate system to the second modality medical image coordinate system. After registration is completed, the optimal spatial transformation parameter is saved for view synchronization. For some non-rigid modality registration requirements, the present application can also be extended to use a nonlinear deformation model to represent T, and to ensure that the calculated deformation field is biologically reasonable by adding regularization constraints.

[0040] S3, a network communication connection is established between the remote consultation participants, the data of the first modality and the second modality medical images are sent to the user terminals of the consultation, and the spatial transformation parameters are synchronized to the user terminals.

[0041] In remote consultation, the image collaborative browsing application of the present application is started at the doctor end and the patient end respectively. Both ends are connected to the collaborative server of the system through the Internet or a medical professional network to establish a secure communication session. Preferably, the communication adopts a client / server architecture, and a Web real-time communication protocol is used to realize low-latency data transmission. First, the initiator (such as the doctor end) sends a consultation request through the collaborative server, including patient identification, medical image sequence identification required for collaboration, etc. After receiving the request, the collaborative server authenticates and notifies the target collaborative party (such as the patient end or another expert end) to join the consultation session. The server locates the corresponding medical image data storage location according to the request, and retrieves the first modality medical image and the second modality medical image from the database or the PACS image storage server. Then, the server sends the target medical image data to the initiator and the collaborative party user terminals through the first communication link (such as a high-speed network channel). After the medical image data transmission is completed, both terminals load the first modality medical image and the second modality medical image locally and prepare for synchronized display. At this point, the remote collaborative session is established, and each end has complete image data and can independently render high-quality images.

[0042] S4, the first user terminal and the second user terminal each display the first modality medical image and the second modality medical image, listen to the image interaction operation on the first user terminal, and convert the interaction operation into a synchronization instruction in real time and send it to the second user terminal through the network;

[0043] S5, the second user terminal receives and executes the synchronization instruction, and updates the medical image displayed locally to make the image view displayed by the second user terminal synchronized with the interaction operation of the first user terminal; wherein if the interaction operation involves positioning a target position on the first modality image, the target position is mapped to the corresponding position on the second modality image using the spatial transformation parameter for synchronized display.

[0044] The system of the application comprises a view synchronization module for listening to the operation of the user on the image locally and sending the operation in the form of a collaborative instruction to another terminal to realize view linkage. Specifically, when the doctor terminal performs interactive operation (such as zooming, panning, adjusting window width and window level, switching tomographic sequence or adding a mark on the image) on the first modality medical image, the local client captures the interactive behavior and generates a corresponding view synchronization instruction. The view synchronization instruction includes necessary parameters, such as: the currently viewed modality type, the image space position (such as the current tomographic sequence number or three-dimensional coordinate slice position), the window parameter and the geometric information of any mark, etc. The instruction is sent in real time to the collaborative server through a second communication link, and then forwarded to the other end collaborative user by the server. After the receiving end client parses the instruction, it automatically performs a synchronous operation on the corresponding image locally. For example, when the doctor terminal scrolls the first modality CT image to the 50th layer section and circles a suspected lesion, the patient terminal receives the instruction, and the local application automatically jumps to the tomography of the corresponding anatomical position of the second modality MRI and displays it with the same zoom ratio, and at the same time, converts the CT coordinates of the doctor's mark into MRI coordinates through the previously calculated registration transformation T* to present the mark at the corresponding position on the MRI image. This intelligent view synchronization ensures that even if the underlying images come from different sources, each end sees the same anatomical region and mark that are spatially aligned, thereby realizing true cross-modality synchronous comparison.

[0045] If there are multiple expert users in the multi-party collaborative consultation, the application supports a dynamic switching mechanism of master control authority: in the initial case, the doctor terminal that initiates controls the image browsing rhythm, and the interfaces of other participants follow the synchronization. When another expert needs to explain, the master control authority can be switched to the expert user by request, and his operation will start to drive the global synchronization, while the original initiator interface changes to passive reception. The system coordinates this authority change through the collaborative server and ensures the consistency of the state of each end (such as the continuity of the currently browsed image index) during the switching process. This mechanism allows multiple experts to take turns operating and explaining on the same set of images, supporting multi-user online collaboration without conflicts.

[0046] S6, when the image mark instruction from another user terminal is received at any user terminal, a corresponding mark is added to the corresponding position of the local image for display, and the mark information is recorded and saved for the consultation report.

[0047] The system of the application further comprises a mark annotation module. The doctor and the remote expert can add region of interest marks (such as arrows, measuring lines, lesion contours or text annotations) on any modality image. These marks are immediately shared to the other end through the view synchronization mechanism and mapped to the correct position of the corresponding modality of the other end by means of the previous multi-modality registration. For example, after the doctor draws a tumor boundary contour on the CT, the MRI image of the patient end will synchronously appear a contour mark covering the corresponding position of the tumor. Both parties can discuss in real time accordingly. All mark data can also be recorded and saved for future review. In addition to image marks, the system preferably integrates audio and video communication or text chat modules, so that the doctor and the patient can communicate while looking at the synchronized images, realizing an immersive consultation experience.

[0048] S7, end the remote collaborative consultation as needed, release the communication connection resources and save the marks and interaction records generated during the consultation after the consultation is completed.

[0049] To improve the performance of real-time collaboration, the application adopts a dual-link architecture to transmit image data and interaction data respectively. That is, after the initial image is transmitted by the server, the subsequent interaction synchronization instructions are transmitted through an independent lightweight channel. In this way, the large volume of image is not repeatedly transmitted every time, effectively reducing the network bandwidth occupation. In addition, the system can adopt data compression and regional incremental update strategies: for example, when the image is translated or zoomed in, only the image block data of the new area entering the field of view is transmitted, thereby further reducing the delay. For dynamic image modalities (such as real-time ultrasound), the system can selectively reduce the frame rate or resolution to synchronize the main information, while still aligning to other modality reference coordinates with high precision. All transmitted data are encrypted and identity authenticated to ensure the security of medical data and the privacy of patients. The system architecture has good fault tolerance, when the network is jittered or temporarily interrupted, the client will cache the recent operations and automatically resynchronize after the connection is restored, ensuring the continuity of the consultation.

[0050] The application further provides an intelligent view synchronization and communication system for multi-modality medical image comparison, which is used for implementing the intelligent view synchronization and communication method for multi-modality medical image comparison as described above, and comprises at least two user terminal devices and a collaborative server in communication connection with the user terminal devices,

[0051] The user terminal device comprises an image display and interaction module for loading and displaying medical images for user browsing and interaction; a view synchronization client module for capturing image interaction operations of a local user and generating synchronization instructions, and receiving remote synchronization instructions and executing them locally to update the image view; the collaboration server comprises an image data storage unit for storing multi-modal medical image data of a patient; a registration calculation unit for calculating spatial transformation parameters between multi-modal medical images; a communication coordination unit for establishing and managing a collaboration session between multiple user terminal devices, and transmitting medical image data and view synchronization instructions between user terminal devices and the server through a network; wherein the spatial transformation parameters calculated by the registration calculation unit are used to guide the view synchronization client module to display views of corresponding positions between different modal medical images, and the communication coordination unit is configured to transmit initial medical image data through a separate first data channel and transmit synchronization instructions through a separate second data channel, so as to reduce data conflicts and improve synchronization real-time performance.

[0052] Embodiments

[0053] The overall architecture of the system is shown in Figure 1 The system of the present embodiment comprises a doctor terminal, a patient terminal, a collaboration server and a network communication link connecting the terminals. The doctor terminal and the patient terminal can be computers, tablets, medical image workstations and other devices with display and interaction capabilities. The collaboration server can be deployed in the cloud or a medical center computer room, and is used for coordinating consultation sessions, storing image data and processing registration calculations. The doctor terminal runs an image display and interaction module and a view synchronization client module, which are used to load medical images, provide user interaction interfaces and send interaction operations to the collaboration server; the patient terminal runs corresponding image display and interaction modules and view synchronization client modules. The collaboration server is provided with an image data storage unit (which can be connected to a hospital PACS system or an image database), a registration calculation unit and a communication coordination unit. The registration calculation unit is used to execute multi-modal medical image registration algorithms to calculate the spatial transformation relationship between different modal medical images. The communication coordination unit is responsible for managing consultation sessions, maintaining terminal connections, and forwarding various data and instructions.

[0054] The application adopts an improved multi-modal registration strategy, which combines mutual information with edge features to improve robustness. In this embodiment, we select MRI T1 sequence as the first modality medical image and CT image as the second modality medical image, and register the data of a brain tumor patient. First, initial alignment is performed: read the DICOM metadata of the two medical images, and roughly align the medical images by using the patient positioning information, such as estimating the initial scaling and translation by comparing the voxel spacing, image center coordinates and the like of CT and MRI. Then enter the iterative optimization stage, taking normalized mutual information as the main similarity index, supplemented by edge overlap constraint. In specific implementation, a multi-resolution pyramid strategy is adopted: first, the images are down-sampled to a lower resolution to calculate mutual information to obtain rough alignment, and then the resolution is gradually increased for fine adjustment. In order to obtain edge indicator functions E A (x) and E B (x), this embodiment extracts edges from CT and MRI images respectively using Canny operator, and takes edge binary graph as structural feature. In the optimization process, mutual information value and edge overlap rate are evaluated simultaneously after updating transformation parameters in each iteration, and convergence is determined according to weighted evaluation function S(T). When the evaluation function is less than the threshold, the iteration is ended. After several rounds of multi-resolution optimization, the registration matrix T* of MRI to CT is finally obtained. After re-slicing the MRI image by applying the matrix, the CT image is well aligned on each section: the brain tissue profile and tumor boundary are highly overlapped under two modalities. Quantitative evaluation shows that the algorithm of the application improves the edge alignment accuracy by about 15% compared with the registration using only mutual information, proving the effectiveness of combining structural information.

[0055] Assume that a specialist doctor uses the doctor terminal, a patient and a local doctor use the patient terminal, and a remote consultation is performed through the system. First, the specialist logs in to the collaboration system on the doctor terminal, initiates a consultation request, specifies the patient ID and the MRI and CT examination sequence numbers to be viewed. After the collaboration server receives the request, it retrieves the corresponding MRI and CT image data sets in the image data storage unit, and notifies the patient terminal that there is a consultation request. After the patient terminal confirms to join the consultation, the collaboration server sends the complete DICOM image data to both terminals through the high-speed link A. After the doctor terminal and the patient terminal each receive the MRI and CT data, they load the images in the local memory. At this time, the registration calculation unit automatically calls the previously pre-calculated MRI-CT registration parameter T* (if not pre-calculated, the registration algorithm is executed in real time) and sends it to the two terminals. Subsequently, enter the synchronous browsing stage: the specialist doctor opens the CT and MRI image windows on the interface of the doctor terminal at the same time, and associates the two windows through the synchronous mode provided by the UI. At this time, the view synchronization client module is enabled, and user operations are monitored. The doctor first scrolls the tomography to the tumor area in the CT window, and adjusts the contrast to clearly display the lesion edge; these operations are captured by the view synchronization client module to form the instruction data "set CT slice index = 50; window width = 300; window level = 40". The instruction is sent to the collaboration server through the link B, and then forwarded to the patient terminal. After the view synchronization client module of the patient terminal parses the instruction, it controls the local image display and interaction module to jump to the 50th layer section and apply the same window parameters to ensure that the CT display of the patient terminal is consistent with that of the doctor terminal. Then, the doctor enables the synchronous linkage function in the MRI window, and when the CT tomography changes, the MRI tomography will be automatically updated according to the registration result. The doctor scrolls the CT image up by 5 layers, and the view synchronization client module captures the change of the CT slice index from 50 to 55, and immediately sends the instruction; the patient terminal synchronizes the CT display to the 55th layer accordingly. At the same time, since the synchronous mode is enabled, the local application of the doctor terminal calculates the MRI layer position corresponding to the 55th layer of CT according to the registration matrix, for example, the 48th layer of MRI, and automatically updates the MRI window of the doctor terminal to this layer. This update also notifies the patient terminal through the instruction, so that the MRI window of the patient terminal is synchronized to the 48th layer section. Through this linkage, when the doctor locates the lesion on one medical image, the other medical image is also automatically switched to the corresponding position, so that the doctor and the patient can see the corresponding lesion performance in CT and MRI on their respective terminals at the same time, realizing true synchronous comparison.

[0056] During the consultation, the two parties communicate the medical condition through integrated audio call. When the doctor circles the lesion range on the medical image, the patient side almost instantaneously sees the same mark appearing on his own screen, and can add other marks or annotations on the MRI image as needed. All these marks are shared with the doctor side through the view synchronization mechanism, thus forming a two-way interaction on the image. For example, the patient's local doctor can mark another suspicious area on the MRI, and the doctor side immediately sees the mark and gives feedback for discussion. Throughout the process, the two parties always communicate around the synchronized image, without the need for repeated language instructions to indicate the position, greatly improving the communication efficiency.

[0057] Figure 3 An interface diagram of multi-modal image synchronous display in a specific embodiment is shown. (a) is the doctor side interface, the left window shows the CT image of the 50th layer section, and the right window shows the section of the same position of the corresponding MRI image; (b) is the patient side interface, which shows the content synchronized with the doctor side. In (a), the lesion mark circled by the doctor on the CT is automatically synchronized and mapped, and the corresponding position is circled with a dashed line on the MRI in (b). The doctor and the patient can intuitively see the difference and corresponding relationship of the lesion under the two modalities.

[0058] The system design of the present application supports the expansion of multiple users and multiple modalities. In an actual application, we deployed the system in a central hospital and multiple primary hospitals to realize multi-party remote image consultation. The collaborative server is configured with higher bandwidth and computing resources, and can accommodate multiple consultations at the same time. Each consultation session may involve more than three image modalities, for example, PET-CT fusion images can also be used as a modal source to access the system for synchronization. For three-dimensional ultrasound or dynamic heart images, the present application can also handle them: due to the large difference between ultrasound image and CT / MRI coordinate system, we use a positioning sensor to obtain the position of the probe in the patient's body coordinate when the patient is undergoing ultrasound examination, and calibrate the real-time ultrasound frame with the pre-acquired CT image. In this way, when the ultrasound image acquires the interested section, the system can calculate the corresponding CT / MRI section position and display it synchronously to the remote expert, which helps the expert to associate the dynamic ultrasound observation with the static CT / MRI data. In the multi-user scenario, we verified the function of switching the control right: in a difficult case discussion, the primary doctor first introduces the CT image performance, then switches the operation control right to the remote expert, the expert immediately explains and marks on the MRI image, and finally another expert switches the control right to make a supplementary explanation on the PET image. Throughout the process, the interfaces of all participants remain synchronized, and the system automatically handles the conflicts of different operators' instructions, ensuring that only the current speaker's operation takes effect and is broadcasted. According to the statistics, the average discussion time is shortened by more than 30% through the collaborative consultation platform, and the consistency of the diagnosis conclusion of remote consultation by hospitals is also significantly improved.

[0059] The above merely describes the preferred embodiments of the present application, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. An intelligent view synchronization and communication method for multi-modal medical image contrast, characterized in that, The method comprises the following steps: obtaining first modality medical images and second modality medical images to be compared, and pre-processing the multi-modality medical images to unify the image format and spatial resolution; performing image registration on the first modality medical images and the second modality medical images to obtain spatial transformation parameters for mapping the first modality image coordinates to the second modality image coordinates; establishing a network communication connection between the remote consultation participants, sending the first modality and second modality medical image data to the user terminals of the consultation, and synchronizing the spatial transformation parameters to the user terminals; displaying the first modality medical images and the second modality medical images on the first user terminal and the second user terminal respectively, listening to the image interaction operations on the first user terminal, and converting the interaction operations into synchronization instructions in real time and sending them to the second user terminal through the network; the second user terminal receives and executes the synchronization instructions, updates the corresponding views of the medical images displayed locally, so that the image views displayed by the second user terminal are synchronized with the interaction operations of the first user terminal; if the interaction operations involve positioning a target position on the first modality image, the target position is mapped to the corresponding position on the second modality image for synchronized display using the spatial transformation parameters; when any user terminal receives an image annotation instruction from another user terminal, a corresponding mark is added to the corresponding position of the local image for display, and the mark information is recorded and saved for the consultation report; ending the remote collaborative consultation as needed, releasing the communication connection resources after the consultation is completed, and saving the marks and interaction records generated during the consultation.

2. The intelligent view synchronization and communication method for multi-modal medical image contrast according to claim 1, characterized in that, The image registration includes initial registration according to the metadata or anatomical landmark points of the medical images, and then iterative solving of the accurate alignment transformation of the multi-modality medical images using an optimization algorithm; the optimization algorithm uses an evaluation function combining statistical intensity similarity and anatomical structure similarity to guide the registration, wherein the evaluation function includes a weighted combination of the mutual information index and the edge overlap index of the images, to improve the robustness and accuracy of the multi-modality image registration.

3. The intelligent view synchronization and communication method for multi-modal medical image contrast according to claim 1, characterized in that, The synchronization instruction includes the image identification, view parameters and spatial transformation parameters involved in the interaction operation; the first user terminal and the second user terminal each locally save complete medical image data, and the execution of the synchronization instruction is realized by re-rendering the image view locally, without the need to repeatedly transmit large-capacity image pixel data.

4. The intelligent view synchronization and communication method for multi-modal medical image contrast according to claim 1, characterized in that, The first user terminal and the second user terminal both support multi-window multi-image synchronous linkage display, and when the first user terminal switches the view linkage mode between the first modality medical images and the second modality medical images, the spatial transformation parameters are used to ensure that the view positions of the two modalities of medical images maintain an anatomical correspondence; when the user scrolls through the medical images in one modality, the medical image window of the other modality automatically jumps to the image at the corresponding anatomical position, realizing cross-modality synchronous comparison and browsing.

5. The intelligent view synchronization and communication method for multi-modal medical image contrast according to claim 1, characterized in that, The method comprises a master control right switching mechanism, initially a user terminal of a party initiating a consultation is a master terminal, and interactive operations thereof are synchronized to all other participating terminals; when another participating terminal requests to obtain the master control right and obtains consent, the switching is performed by the requesting terminal as a new master terminal, and subsequent interactive operations are synchronized to the remaining terminals, and the original master terminal becomes a passive synchronization state. The switching of the master control right is coordinated by the collaborative server to ensure continuous consistency of the image view state of each terminal and uninterrupted synchronization.

6. An intelligent view synchronization and communication system for multi-modal medical image contrast, characterized in that, The method comprises at least two user terminal devices and a collaborative server in communication connection with the user terminal devices, the user terminal devices comprising an image display and interaction module for loading and displaying medical images for user browsing and interaction; a view synchronization client module for capturing image interaction operations of a local user and generating synchronization instructions, and receiving remote synchronization instructions and executing locally to update the image view. The collaborative server comprises an image data storage unit for storing multi-modal medical image data of a patient; a registration calculation unit for calculating spatial transformation parameters between multi-modal medical images; a communication coordination unit for establishing and managing a collaborative session between multiple user terminal devices, and transmitting medical image data and view synchronization instructions between user terminal devices and the server through a network; wherein the spatial transformation parameters calculated by the registration calculation unit are used to guide the view synchronization client module to display views at corresponding positions between different modal medical images, and the communication coordination unit is configured to transmit initial medical image data through an independent first data channel and transmit synchronization instructions through an independent second data channel, so as to reduce data conflicts and improve synchronization real-time performance.

7. The intelligent view synchronization and communication system for multi-modal medical image contrast according to claim 6, wherein, The registration calculation unit is configured to automatically align multi-modal medical images by using a registration algorithm based on mutual information and edge feature fusion, and transmit the obtained spatial transformation matrix to each user terminal device; the view synchronization client module converts a coordinate point on one modal medical image into a corresponding coordinate point on another modal medical image according to the spatial transformation matrix, so as to synchronously display anatomically corresponding markers or cursor positions in different modal medical image views of each user terminal.

8. The intelligent view synchronization and communication system for multi-modal medical image contrast according to claim 6, wherein, The communication coordination unit further comprises a right management component for switching the master control right of view synchronization between multiple user terminal devices, so that the interactive operation of the user terminal device currently having the master control right is broadcasted to other user terminal devices, and the consistency of the medical image display state of each user terminal device is maintained when the master control right is switched.

9. The intelligent view synchronization and communication system for multi-modal medical image comparison according to claim 6, wherein, The user terminal device further comprises an annotation marking module configured to accept graphical or text marks added to the medical image by the user; the view synchronization client module sends the marks added by one user terminal to other user terminals through the communication coordination unit, and displays the marks on the corresponding modality medical image of the other user terminals by mapping the marks to the correct positions of the medical image using the spatial transformation parameters, thereby realizing real-time sharing of the mark information; the annotation marking module is further configured to save all the marks for viewing after the consultation.

10. The intelligent view synchronization and communication system for multi-modal medical image contrast according to claim 6, wherein, The communication coordination unit uses an encryption algorithm to protect the transmitted medical image data and synchronization instruction data, and performs identity authentication on the connected user terminal devices, so as to ensure patient privacy and data security; the system has a reconnection function after network interruption, and can automatically resend the key synchronization instructions not executed during the interruption after the connection is restored, thereby ensuring the integrity of the remote collaboration process.

Images