Vision-based three-dimensional medical image diagnosis method, system, medium, terminal and program product

Abstract

The application provides a visual-based three-dimensional medical image diagnosis method, system, medium, terminal and program product, comprising: acquiring a medical image video in a rolling medical image slice browsing process recorded by a camera; positioning the medical image video frame by frame based on a two-level positioning mechanism and cutting out a medical image region image; inputting the medical image region image into a medical visual model to extract a medical image feature vector to constitute a medical image feature sequence; inputting the medical image feature sequence into a sequence modeling model to capture an inter-frame context relationship and output a frame-level feature sequence; configuring a task query vector for each diagnosis task, and generating a task feature vector based on a hybrid expert model; and inputting the task feature vector into a multi-task classifier to output a diagnosis result. The application can recover three-dimensional semantic information from a screen browsing video and complete multi-lesion diagnosis without accessing a PACS system or obtaining original DICOM files.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of medical imaging artificial intelligence (AI) and computer vision (CVR), and in particular to a vision-based three-dimensional medical image diagnosis method, system, medium, terminal, and program product. Background Technology

[0002] Medical imaging diagnosis is one of the most critical links in the modern healthcare system, especially in the screening and diagnosis of diseases of the chest, abdomen, and nervous system. Three-dimensional computed tomography (CT) and magnetic resonance imaging (MRI) have become clinical standards. Traditional medical imaging artificial intelligence (AI) systems typically rely on high-quality, non-destructive DICOM (Digital Imaging and Communications in Medicine) data as input, which is centrally managed by the hospital's PACS (Picture Archiving and Communication System). Because DICOM files contain complete voxel information, imaging parameters, and metadata, they have long been considered a prerequisite for reliable computer-aided diagnosis (CAD). However, direct access to the PACS system often requires complex network permissions, data interfaces, compliance approvals, and IT support, making AI systems difficult to deploy in many healthcare institutions, especially in primary care hospitals, low-resource areas, or healthcare environments that have not yet undergone digital transformation.

[0003] With the widespread use of mobile devices and high-resolution cameras, clinicians frequently record screen content using their phones or external cameras for remote consultations, teaching discussions, or case recording. This type of screen recording has extremely low deployment costs, does not rely on PACS interfaces, and does not require changes to existing hospital workflows, thus possessing natural scalability. However, screen videos recorded by cameras differ significantly from raw DICOM data, including reduced resolution, Moiré stripes, compression artifacts, reflections, motion blur, uneven scrolling speed, and frame loss, among other degradation factors. Furthermore, browsing 3D images is inherently a dynamic process; doctors observe volumetric structures by scrolling through slices, while cameras record a time-series video where each frame does not correspond one-to-one with the original volumetric slices. These factors prevent traditional DICOM-based 3D AI models from being directly applied to camera-recorded screen videos.

[0004] Therefore, it is necessary to provide a vision-based three-dimensional medical image diagnostic method, system, medium, terminal, and program product to solve the above-mentioned problems in the prior art. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a vision-based three-dimensional medical image diagnosis method, system, medium, terminal and program product to solve the technical problem that existing medical image artificial intelligence systems must rely on high-quality DICOM data and cannot perform three-dimensional diagnosis from screen browsing videos recorded by cameras.

[0006] To achieve the above and other related objectives, the first aspect of this application provides a vision-based three-dimensional medical image diagnosis method, comprising: acquiring medical image videos during the three-dimensional image browsing process, recorded by a camera and showing the screen content of scrolling three-dimensional medical image slices in medical image browsing software; performing frame-by-frame localization of the acquired medical image videos based on a preset two-level localization mechanism to obtain medical viewport regions, and cropping medical image region images from the localized medical viewport regions; inputting the cropped medical image region images into a pre-trained medical vision model to extract medical image feature vectors corresponding to each video frame, thereby forming a medical image feature sequence; inputting the medical image feature sequence into a pre-constructed sequence modeling model to capture the contextual relationships between each video frame, and outputting a context-enhanced frame-level feature sequence; configuring a corresponding task query vector for each diagnostic task, and generating task feature vectors corresponding to each diagnostic task based on a hybrid expert model according to the task query vectors and frame-level feature sequences corresponding to each diagnostic task; and inputting the task feature vectors corresponding to each diagnostic task into a multi-task classifier to output the diagnostic results corresponding to each diagnostic task.

[0007] In some embodiments of the first aspect of this application, the step of performing frame-by-frame positioning of the acquired medical image video based on a preset two-level positioning mechanism to obtain a medical viewing window region, and cropping a medical image region image from the positioned medical viewing window region, specifically includes: performing first-level positioning on each video frame of the medical image video based on a first object detection model to locate the display area, and correcting the display area based on perspective transformation technology to obtain a standard rectangular display image; performing second-level positioning on the rectangular display image based on a second object detection model to locate the medical viewing window region, and cropping the medical viewing window region to obtain a medical image region image; scaling the cropped medical image region image to a preset uniform size to convert it into an input size suitable for the medical vision model input.

[0008] In some embodiments of the first aspect of this application, the step of generating task feature vectors corresponding to each diagnostic task based on a hybrid expert model according to the task query vectors and frame-level feature sequences corresponding to each diagnostic task specifically includes: selecting frame regions associated with each diagnostic task from the frame-level feature sequence based on the task query vectors corresponding to each diagnostic task using a cross-attention mechanism, so as to output preliminary aggregated feature vectors for each diagnostic task; inputting the preliminary aggregated feature vectors into the gate network of the hybrid expert model to dynamically select the expert combination of the hybrid expert model; and weightedly fusing the feature vectors output by each expert in the expert combination of the hybrid expert model to obtain the task feature vectors corresponding to each diagnostic task.

[0009] In some embodiments of the first aspect of this application, the medical vision model is pre-trained based on a label-free self-distillation model.

[0010] In some embodiments of the first aspect of this application, the sequence modeling model is pre-built based on the Transformer model.

[0011] In some embodiments of the first aspect of this application, the architecture of the multi-task classifier includes a fully connected layer and an activation function layer.

[0012] To achieve the above and other related objectives, a second aspect of this application provides a vision-based three-dimensional medical image diagnostic system, comprising: a video acquisition module for acquiring medical image videos recorded by a camera during the three-dimensional image browsing process, where the screen content of scrolling three-dimensional medical image slices in medical image browsing software is displayed; a two-level positioning module for performing frame-by-frame positioning of the acquired medical image video based on a preset two-level positioning mechanism to obtain medical viewport regions, and cropping medical image region images from the located medical viewport regions; and a feature extraction module for inputting the cropped medical image region images into a pre-trained medical vision model to extract the corresponding features of each video frame. The system uses medical image feature vectors to form a medical image feature sequence; a sequence modeling module to input the medical image feature sequence into a pre-built sequence modeling model to capture the contextual relationships between video frames and output a context-enhanced frame-level feature sequence; a task-specific aggregation module to configure a corresponding task query vector for each diagnostic task and generate task feature vectors for each diagnostic task based on the task query vectors and frame-level feature sequences corresponding to each diagnostic task using a hybrid expert model; and a multi-lesion diagnostic output module to input the task feature vectors corresponding to each diagnostic task into a multi-task classifier to output the diagnostic results for each diagnostic task.

[0013] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0014] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the method.

[0015] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0016] As described above, the vision-based three-dimensional medical image diagnostic method, system, medium, terminal, and program product of this application have the following beneficial effects:

[0017] First, the system acquires medical image videos during the 3D image browsing process, captured by a camera and showing the screen content of scrolling 3D medical image slices in medical image browsing software. Then, based on a pre-set two-level localization mechanism, the acquired medical image video is localized frame-by-frame to obtain medical viewport regions, and medical image regions are cropped from these regions. The cropped medical image regions are then input into a pre-trained medical vision model to extract medical image feature vectors corresponding to each video frame, forming a medical image feature sequence. Finally, the medical image feature sequence is input into a pre-constructed sequence modeling model to capture the contextual relationships between video frames and output context-enhanced frame-level data. The system first generates a feature sequence; then, it configures a corresponding task query vector for each diagnostic task, and generates a task feature vector for each diagnostic task based on the task query vector and frame-level feature sequence of each diagnostic task using a hybrid expert model; finally, it inputs the task feature vector of each diagnostic task into a multi-task classifier to output the diagnostic results of each diagnostic task. This enables the recovery of three-dimensional semantic information and the completion of multi-lesion diagnosis from screen-viewed videos without accessing the PACS system or obtaining the original DICOM file. At the same time, this application significantly reduces the deployment threshold of medical imaging AI systems, making them suitable for low-resource hospitals, primary healthcare institutions, and scenarios where PACS cannot be upgraded. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart of a vision-based three-dimensional medical image diagnosis method according to one embodiment of this application.

[0019] Figure 2 The diagram shown illustrates the working principle of a vision-based three-dimensional medical image diagnosis method in one embodiment of this application.

[0020] Figure 3 The diagram shown is a block diagram of a vision-based three-dimensional medical image diagnostic system according to an embodiment of this application.

[0021] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0022] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0023] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0024] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0025] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0026] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0027] <1> Mixture of Experts (MoE) is a technique to improve the quality of large language models (LLMs). It achieves better performance by introducing multiple different sub-models (i.e., experts). Its structure mainly consists of two core components: experts and routers or gate networks. Here, experts refer to the fact that each layer of the feedforward neural network (FFNN) is no longer a single structure, but is composed of multiple "experts", and each "expert" is essentially a feedforward neural network. The router or gate network is responsible for deciding which experts should process each token.

[0028] <2> Perspective transformation is a technique that projects an image onto a new viewing plane, also known as projective mapping. It is a non-linear transformation that maps a point in a two-dimensional coordinate system to a point in a three-dimensional coordinate system, and then projects it onto a point in another two-dimensional coordinate system. Perspective transformation can change the shape in an image and can simulate the perspective effect in the real world.

[0029] <3> Cross-Attention is a variant of the attention mechanism used to establish a connection between two different input sequences and compute attention weights.

[0030] <4> The self-distillation with no labels (DINO) model is a self-supervised learning method that has achieved remarkable results in the image domain. Its core idea is to allow a "student network" to learn the representation of a "teacher network" on unlabeled data through knowledge distillation. Its key innovation lies in the fact that the parameters of the teacher network are the exponential moving average (EMA) of the parameters of the student network. Furthermore, by avoiding the use of contrastive learning with negative sample pairs, it avoids the need for a large negative sample memory, thus achieving efficient and powerful visual representation learning.

[0031] <5> The Transformer model is a revolutionary deep learning model, particularly popular in Natural Language Processing (NLP). Its core innovation is the self-attention mechanism, which allows the model to consider all positions in a sequence while processing sequential data. It consists of two main parts: an encoder and a decoder. The encoder is composed of multiple identical layers stacked together, each containing two sub-layers: a multi-head attention mechanism and a simple, position-wise fully connected feed-forward network. The decoder is also composed of multiple identical layers stacked together, but each layer has three sub-layers, including a masked multi-head attention mechanism, an encoder-decoder attention mechanism, and a feed-forward network.

[0032] <6> Fully connected layer: This is a layer in a neural network where each neuron is connected to all neurons in the previous layer. It is mainly used to map features to the output space to achieve classification or regression tasks.

[0033] <7> The Sigmoid function is an S-shaped activation function that maps the input to a value between 0 and 1. It is often used for output layer probability prediction in binary classification problems.

[0034] <8> Azure Kinect is an advanced development kit designed for developers and businesses. It includes multiple high-performance sensors and cameras that can capture and process spatial data, supporting the construction of complex computer vision and speech models.

[0035] <9> Real-time Models for Object Detection (RTMDet) is a high-performance real-time object detector designed to surpass the performance of the YOLO series and extend to tasks such as instance segmentation and rotated object detection. It achieves an excellent balance between speed and accuracy through innovative model architecture, dynamic label assignment strategy and optimized training methods.

[0036] <10> YOLO (You Only Look Once) model: It is an object detection algorithm in the field of computer vision. This algorithm treats object detection as a single regression problem and can complete the detection through a single forward propagation, which has a fast processing speed.

[0037] In existing technologies, some exploration has been conducted on the diagnosis of two-dimensional medical images (such as X-rays) captured by cameras. Studies have shown that some diagnostic information can be recovered through geometric correction and image enhancement. However, these methods cannot be directly extended to three-dimensional images because the diagnosis of three-dimensional images depends on the continuity of volumetric structure and the spatial relationships between slices, while frame sequences in camera videos often suffer from problems such as repetition, frame skipping, and inconsistent order. In addition, traditional 3D Convolutional Neural Networks (3D CNNs) or voxel-based Transformer models require input as regular three-dimensional tensors, which cannot handle video sequences of variable length and significant inter-frame differences. Even if the video is simply converted into an image sequence and input into existing models, the lack of volumetric consistency leads to a severe decline in diagnostic performance. Meanwhile, Vision Foundation Models (VFMs) that have emerged in recent years have demonstrated strong semantic understanding capabilities in both natural and medical images. These models typically achieve robustness to low-level visual degradation through large-scale self-supervised learning, and can still extract stable high-level semantic features even when images are noisy, blurred, or compressed. However, there is still a lack of effective methods for applying these foundational models to camera-recorded videos of three-dimensional medical images. Especially in diagnostic scenarios involving multiple lesions and multiple tasks, different lesions have different spatial distributions in the volume. The model needs to have task-specific feature aggregation capabilities, but most existing methods use simple average pooling or attention pooling, which cannot make full use of the structural information in the video sequence.

[0038] Therefore, existing technologies have significant shortcomings in the following aspects: First, they lack diagnostic methods capable of recovering 3D semantic information from screen browsing videos recorded by cameras; second, they lack robust medical image feature extraction mechanisms for severe video domain degradation; third, they lack model structures capable of processing variable-length video sequences and achieving task-specific aggregation; and fourth, they lack a general-purpose medical AI diagnostic framework that can be deployed without a PACS interface or access to DICOM files. To address these issues, this application provides a vision-based 3D medical image diagnostic method, system, medium, terminal, and program product, enabling the constructed medical image AI system to directly diagnose 3D medical images from video recorded by cameras. This overcomes the limitations of traditional PACS integration, significantly reduces the deployment cost of medical image AI systems, and improves the global accessibility of medical image AI systems.

[0039] To facilitate understanding of the embodiments of this application, in conjunction with Figure 1 and Figure 2 Detailed explanation. Figure 1 A flowchart illustrating a vision-based three-dimensional medical image diagnosis method according to an embodiment of the present invention is shown. Figure 2 This illustration shows a schematic diagram illustrating the working principle of a vision-based three-dimensional medical image diagnosis method according to an embodiment of the present invention. The vision-based three-dimensional medical image diagnosis method in this embodiment includes the following steps:

[0040] Step S11: Acquire the medical image video recorded by the camera during the 3D image browsing process, which is presented by scrolling the screen content of the 3D medical image slices in the medical image browsing software.

[0041] By using an external camera to record the screen content of medical image viewing software, medical image videos containing the slide scrolling process can be obtained.

[0042] Specifically, an external RGB camera is fixed in front of the monitor of a terminal device with medical image viewing software installed. The camera can be any device with high-definition shooting capabilities, such as Azure Kinect, a mobile phone camera, or an industrial camera. The distance between the camera and the monitor is set to 0.5–0.6 meters, and the shooting angle can be adjusted within ±15° to simulate the viewing posture of a doctor in a real clinical environment. Simultaneously, the camera records screen content at a frame rate of 30 frames per second, with a resolution of 1920×1080 or 2560×1440.

[0043] Furthermore, during the acquisition process, doctors or automated programs scroll through slices of 3D medical images (e.g., CT or MRI) in medical image viewing software (such as ITK-SNAP, RadiAnt, Horos, etc.). The scrolling speed can be non-uniform, potentially including rapid sliding, short pauses, and reverse scrolling. Moreover, due to the randomness of the scrolling behavior, the same slice may correspond to multiple video frames, and there may be frame skipping, repeated frames, or inconsistent frame intervals. Since subsequent steps can automatically handle these irregularities, this step does not require manual annotation or slice alignment of the video. Thus, video-based acquisition of the 3D medical image viewing process based on camera screen recording is achieved.

[0044] Step S12: Based on the preset two-level positioning mechanism, the acquired medical image video is positioned frame by frame to obtain the medical window region, and the medical image region image is cropped from the positioned medical window region.

[0045] By using a pre-defined two-level positioning mechanism, namely Screen Viewport Localization (SVL) technology, video frames are positioned at two levels, automatically cropping out the medical image area, and eliminating perspective distortion caused by the shooting angle through geometric correction.

[0046] In some embodiments of this application, the step of performing frame-by-frame localization of the acquired medical image video based on a preset two-level localization mechanism to obtain a medical viewing window region, and cropping a medical image region image from the localized medical viewing window region, specifically includes: performing first-level localization on each video frame of the medical image video based on a first object detection model to locate the display area, and correcting the display area based on perspective transformation technology to obtain a standard rectangular display image; performing second-level localization on the rectangular display image based on a second object detection model to locate the medical viewing window region, and cropping the medical viewing window region to obtain a medical image region image; scaling the cropped medical image region image to a preset uniform size to convert it into an input size suitable for the medical vision model input.

[0047] By designing a pre-defined two-level localization mechanism, screen detection and medical window localization are achieved, thereby automatically extracting medical image regions from video recorded by a camera. Specifically, in the first level of localization, a first object detection model, such as the RTMDet model, YOLOv8 model, or other lightweight detection models, is used to identify the display region in the video frame. The detection result is a quadrilateral or rectangular bounding box. Further, since the camera shooting angle may be tilted, perspective transformation technology is used to further perform perspective transformation on the detected display region to correct it into a standard rectangular image, eliminating geometric distortion caused by the shooting angle, and finally obtaining a standard rectangular display image. In the second level of localization, a second object detection model, such as the YOLOv5 model or YOLOv8 model, is used to identify the medical window region in the medical image browsing software. The medical window region typically contains medical image (e.g., CT or MRI) slices, but does not contain interface elements such as toolbars, menu bars, or patient information bars. It should be noted that the second object detection model can be trained on more than 10,000 labeled frames to adapt to different software interfaces, different display sizes, and different lighting conditions.

[0048] After the two-level localization is completed, the medical image region image, i.e., N video slices, is cropped from the medical window region and scaled to a uniform size, such as 512×512 pixels, so as to convert it into an input format suitable for medical vision model input for subsequent feature extraction.

[0049] Step S13: Input the cropped medical image region image into the pre-trained medical vision model to extract the medical image feature vectors corresponding to each video frame, so as to form a medical image feature sequence.

[0050] By inputting the cropped medical image region image frames into a pre-trained medical vision model (image encoder), the medical image feature vectors corresponding to each video frame with semantic robustness are extracted, thereby forming a medical image feature sequence.

[0051] In some embodiments of this application, the medical vision model is pre-trained based on a label-free self-distillation model.

[0052] Specifically, the cropped medical image region frames are input into a pre-trained medical vision model. This model is pre-trained based on an unlabeled self-distillation model, such as DINOv2 or DINOv3, to extract semantically robust and stable features. The medical vision model achieves robustness against degradation factors such as noise, blur, compression artifacts, and reflections through large-scale self-supervised learning. Furthermore, the input size of the medical vision model can be 512×512, and the output is a C-dimensional (e.g., 768-dimensional or 1024-dimensional) medical image feature vector.

[0053] To maintain the generalization ability of the medical vision model, its parameters are kept frozen and no fine-tuning is performed on the camera video. This ensures that the medical vision model can still extract stable semantic features under different devices, different lighting conditions, and different software interfaces. Each video frame is independently encoded to form a medical image feature sequence of length N, where N is the number of video frames, usually between 200 and 800.

[0054] Step S14: Input the medical image feature sequence into the pre-built sequence modeling model to capture the contextual relationship between each video frame, and output the context-enhanced frame-level feature sequence.

[0055] By modeling variable-length video sequences using a sequence modeling model, the contextual relationships between slices can be captured, generating context-enhanced frame-level feature sequences.

[0056] In some embodiments of this application, the sequence modeling model is pre-built based on the Transformer model.

[0057] Specifically, since there is no strict correspondence between video frames and original slices, it is necessary to recover the continuity of the 3D structure from the frame sequence. To address this, this embodiment pre-builds a sequence modeling model based on the Transformer model, which includes two layers of self-attention encoders, each containing eight attention heads.

[0058] The constructed sequence modeling model can capture the contextual relationships between frames. For example, subtle changes between consecutive frames reflect the slice scrolling direction, pause frames correspond to key anatomical structures, and feature repetition is caused by reverse scrolling. The output of the sequence modeling model is a context-enhanced frame-level feature sequence, which provides the basis for subsequent task-specific aggregation steps, thereby enabling the modeling of variable-length video sequences.

[0059] Step S15: Configure a corresponding task query vector for each diagnostic task, and generate task feature vectors for each diagnostic task based on the hybrid expert model according to the task query vectors and frame-level feature sequences corresponding to each diagnostic task.

[0060] By introducing a task-specific aggregation mechanism for multi-lesion diagnosis, namely, designing task query vectors and hybrid expert models, the differential feature extraction and aggregation of different lesions can be achieved.

[0061] In some embodiments of this application, the step of generating task feature vectors corresponding to each diagnostic task based on the task query vectors and frame-level feature sequences corresponding to each diagnostic task specifically includes: selecting frame regions associated with each diagnostic task from the frame-level feature sequence based on the task query vectors corresponding to each diagnostic task using a cross-attention mechanism, so as to output preliminary aggregated feature vectors for each diagnostic task; inputting the preliminary aggregated feature vectors into the gate network of the hybrid expert model to dynamically select the expert combination of the hybrid expert model; and weightedly fusing the feature vectors output by each expert in the expert combination of the hybrid expert model to obtain the task feature vectors corresponding to each diagnostic task.

[0062] Specifically, firstly, a learnable task query vector is set for each diagnostic task (e.g., metal implant detection, pleural effusion detection, etc.), resulting in a total of M task query vectors. Each task query vector is a C-dimensional learnable vector. Then, the task query vectors select frame regions related to each diagnostic task from frame-level sequence features through a cross-attention mechanism, outputting a preliminary aggregated feature vector for each diagnostic task. For example, metal implants typically appear in high-density areas, while pleural effusion appears in low-contrast soft tissue areas.

[0063] Then, to further enhance task specificity, a Mixture-of-Experts (MoE) model is introduced, specifically designed as a Task-Aware Multi-gate Mixture-of-Experts (Task MMoE) model. This model contains K experts, each executing a different feature aggregation strategy, such as density-based attention, texture-based attention, location-based attention, and global context-based attention. The gating network corresponding to each diagnostic task generates expert weights based on the initial aggregated feature vector input. Then, based on the dynamically selected expert combination, each expert executes its corresponding feature aggregation strategy according to the initial aggregated feature vector (token weight), performing a linear combination of sliced ​​features in the input frame-level feature sequence to achieve differentiated aggregation. Finally, the feature vectors output by each expert in the expert combination are weighted and fused to obtain the task feature vectors (i.e., the output linear combination) corresponding to each diagnostic task. The final output task feature vectors are used for subsequent multi-task classifier diagnosis.

[0064] Step S16: Input the task feature vectors corresponding to each diagnostic task into the multi-task classifier to output the diagnostic results corresponding to each diagnostic task.

[0065] By designing a multi-task classifier, the diagnostic probabilities of multiple lesions are output based on the task feature vectors corresponding to each diagnostic task, thereby realizing multi-task three-dimensional diagnosis, such as outputting medical material detection results and pleural effusion detection results.

[0066] In some embodiments of this application, the architecture of the multi-task classifier includes a fully connected layer and an activation function layer.

[0067] Specifically, the aggregated task feature vectors are input into a multi-task classifier. Each diagnostic task corresponds to an independent binary classifier head. The multi-task classifier can employ fully connected layers plus activation function layers, such as the sigmoid function, to output the probability value of each lesion. It should be noted that this embodiment can be expanded to include more tasks as needed, such as adding tasks for lung nodule detection, atelectasis detection, and pericardial effusion detection. The final diagnostic results can be displayed through a graphical interface, including: the probability of each lesion, a task-specific attention heatmap, and keyframe localization results. This information helps doctors understand the model's decision-making basis and improves the system's interpretability.

[0068] The vision-based 3D medical image diagnosis method provided in this application has the following beneficial effects: First, it realizes 3D medical image diagnosis from screen browsing videos recorded by a camera for the first time, breaking through the limitation that traditional medical image AI systems must rely on DICOM data; Second, through two-level localization and semantically robust feature extraction, it effectively overcomes degradation factors such as noise, blur, compression artifacts, and reflections in camera screen recording videos; Third, the task-specific aggregation mechanism proposed in this application can automatically select key frame regions for different lesions, improving diagnostic accuracy and interpretability; Fourth, this application does not require access to a PACS system, has extremely low deployment costs, is suitable for resource-constrained medical environments, and has significant promotional value.

[0069] Figure 3 This is a schematic block diagram of a vision-based three-dimensional medical image diagnostic system provided in an embodiment of this application. Figure 3 As shown, the vision-based three-dimensional medical image diagnostic system 300 includes:

[0070] The video acquisition module 301 is used to acquire medical image videos recorded by the camera during the three-dimensional image browsing process, which are presented by the screen content of scrolling three-dimensional medical image slices in medical image browsing software.

[0071] The two-level positioning module 302 is used to perform frame-by-frame positioning of the acquired medical image video based on a preset two-level positioning mechanism to obtain the medical window region, and to crop out the medical image region image from the positioned medical window region.

[0072] The feature extraction module 303 is used to input the cropped medical image region image into a pre-trained medical vision model to extract the medical image feature vectors corresponding to each video frame, so as to form a medical image feature sequence.

[0073] The sequence modeling module 304 is used to input medical image feature sequences into a pre-built sequence modeling model to capture the contextual relationships between video frames and output context-enhanced frame-level feature sequences.

[0074] The task-specific aggregation module 305 is used to configure a corresponding task query vector for each diagnostic task, and generate task feature vectors for each diagnostic task based on a hybrid expert model according to the task query vectors and frame-level feature sequences corresponding to each diagnostic task.

[0075] The multi-lesion diagnostic output module 306 is used to input the task feature vectors corresponding to each diagnostic task into the multi-task classifier to output the diagnostic results corresponding to each diagnostic task.

[0076] The vision-based 3D medical imaging diagnostic system provided in this application can be deployed on a regular computer without accessing a PACS system or installing a DICOM parsing library. Doctors only need to record the screen using a camera to complete the 3D diagnosis. Therefore, the vision-based 3D medical imaging diagnostic system is particularly suitable for scenarios where primary hospitals cannot access PACS, remote consultations, teaching demonstrations, rapid AI integration with medical equipment manufacturers, and medical image-assisted diagnosis in resource-poor countries. Furthermore, the vision-based 3D medical imaging diagnostic system has low hardware requirements and extremely low deployment costs, making it highly valuable for widespread adoption.

[0077] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0078] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0079] Figure 4 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 4 As shown, the electronic terminal 400 includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic terminal 400 are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0080] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0081] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0082] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include: any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application programs 4022; the operating system 4021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may include various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention can be included in the application program 4022.

[0083] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0084] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0085] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figures 1 to 2 The method of any of the embodiments shown.

[0086] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figures 1 to 2 The method of any of the embodiments shown.

[0087] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0088] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0089] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0093] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0094] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] In summary, addressing the technical problem that existing medical imaging AI systems rely on high-quality DICOM data and cannot perform 3D diagnosis from screen browsing videos recorded by cameras, this application provides a vision-based 3D medical image diagnosis method, system, medium, terminal, and program product. This method achieves 3D medical image diagnosis from screen browsing videos recorded by cameras without direct access to DICOM data. It involves medical image video acquisition technology, screen and window positioning technology, semantically robust feature extraction of medical vision models, multi-task specific aggregation mechanisms, and a diagnostic framework for low-resource medical scenarios that does not rely on image archiving and communication systems (PACS-agnostic). Specifically, it involves first acquiring medical image videos during the 3D image browsing process, presented by the screen content of scrolling 3D medical image slices in medical image browsing software, recorded by a camera; then, based on a preset two-level positioning mechanism, performing frame-by-frame positioning of the acquired medical image videos. The process involves obtaining a medical viewpoint region and cropping a medical image region from it. This cropped image region is then input into a pre-trained medical vision model to extract medical image feature vectors for each video frame, forming a medical image feature sequence. This sequence is then input into a pre-built sequence modeling model to capture the contextual relationships between video frames and output a context-enhanced frame-level feature sequence. Next, a corresponding task query vector is configured for each diagnostic task, and based on the task query vector and frame-level feature sequence, a hybrid expert model generates task feature vectors for each diagnostic task. Finally, the task feature vectors for each diagnostic task are input into a multi-task classifier to output the diagnostic results. This allows for the recovery of 3D semantic information and multi-lesion diagnosis from screen-viewed videos without accessing a PACS system or obtaining the original DICOM file. Furthermore, this application significantly lowers the deployment threshold for medical imaging AI systems, making them suitable for low-resource hospitals, primary healthcare institutions, and scenarios where PACS upgrades are not feasible. Therefore, this application effectively overcomes the shortcomings of existing technologies and possesses high industrial applicability.

[0096] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A vision-based three-dimensional medical image diagnostic method, characterized in that, include: Acquire medical image videos during the 3D image browsing process, which are captured by a camera and show the screen content of scrolling 3D medical image slices in medical image browsing software. Based on a preset two-level positioning mechanism, the acquired medical image video is positioned frame by frame to obtain the medical window region, and the medical image region image is cropped from the positioned medical window region. The cropped medical image region is input into a pre-trained medical vision model to extract the medical image feature vectors corresponding to each video frame, thus forming a medical image feature sequence. The medical image feature sequence is input into a pre-built sequence modeling model to capture the contextual relationships between video frames and output a context-enhanced frame-level feature sequence. Configure a corresponding task query vector for each diagnostic task, and generate task feature vectors for each diagnostic task based on the hybrid expert model according to the task query vectors and frame-level feature sequences corresponding to each diagnostic task. The task feature vectors corresponding to each diagnostic task are input into a multi-task classifier to output the diagnostic results corresponding to each diagnostic task.

2. The vision-based three-dimensional medical image diagnostic method according to claim 1, characterized in that, The method of performing frame-by-frame positioning on the acquired medical image video based on a preset two-level positioning mechanism to obtain the medical view area, and then cropping the medical image region image from the located medical view area, specifically includes: The first-level localization of each video frame of the medical image video is performed based on the first target detection model to locate the display area, and the display area is corrected based on the perspective transformation technology to obtain a standard rectangular display image. The second target detection model is used to perform second-level localization on the rectangular display image to locate the medical window region, and the medical window region is cropped to obtain the medical image region image. The cropped medical image region is scaled to a preset uniform size to convert it into an input size suitable for medical vision model input.

3. The vision-based three-dimensional medical image diagnostic method according to claim 1, characterized in that, The step of generating task feature vectors for each diagnostic task based on the task query vectors and frame-level feature sequences corresponding to each diagnostic task, using a hybrid expert model, specifically includes: Based on the cross-attention mechanism, the frame regions associated with each diagnostic task are selected from the frame-level feature sequence according to the task query vector corresponding to each diagnostic task, so as to output the preliminary aggregated feature vector of each diagnostic task. The initial aggregated feature vector is input into the gate network of the hybrid expert model to dynamically select the expert combination of the hybrid expert model; The feature vectors output by each expert in the expert combination of the hybrid expert model are weighted and fused to obtain the task feature vector corresponding to each diagnostic task.

4. The vision-based three-dimensional medical image diagnosis method according to claim 1, characterized in that, The medical vision model is pre-trained based on a label-free self-distillation model.

5. The vision-based three-dimensional medical image diagnostic method according to claim 1, characterized in that, The sequence modeling model is pre-built based on the Transformer model.

6. The vision-based three-dimensional medical image diagnostic method according to claim 1, characterized in that, The architecture of the multi-task classifier includes a fully connected layer and an activation function layer.

7. A vision-based three-dimensional medical image diagnostic system, characterized in that, include: The video acquisition module is used to acquire medical image videos recorded by the camera during the 3D image browsing process, which are presented by the screen content of scrolling 3D medical image slices in medical image browsing software. The two-level positioning module is used to perform frame-by-frame positioning of the acquired medical image video based on a preset two-level positioning mechanism to obtain the medical window region, and to crop out the medical image region image from the located medical window region. The feature extraction module is used to input the cropped medical image region image into a pre-trained medical vision model to extract the medical image feature vectors corresponding to each video frame, so as to form a medical image feature sequence. The sequence modeling module is used to input medical image feature sequences into a pre-built sequence modeling model to capture the contextual relationships between video frames and output context-enhanced frame-level feature sequences. The task-specific aggregation module is used to configure a corresponding task query vector for each diagnostic task, and generate task feature vectors for each diagnostic task based on a hybrid expert model according to the task query vectors and frame-level feature sequences corresponding to each diagnostic task. The multi-lesion diagnosis output module is used to input the task feature vectors corresponding to each diagnostic task into the multi-task classifier to output the diagnostic results corresponding to each diagnostic task.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method as described in any one of claims 1 to 6.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1 to 6.

Images