Medical image processing method, device, equipment and storage medium

Abstract

The present application relates to a medical image processing method, apparatus, device, and storage medium. The method comprises: extracting surface sampling points of a region of interest from a medical image to be processed; constructing a region of interest map representation based on the surface sampling points of the region of interest; extracting multiple modal features corresponding to each surface sampling point in the region of interest map representation; fusing the multiple modal features corresponding to all surface sampling points in the region of interest map representation to obtain a target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest. The present method can fuse multiple modal features of surface sampling points to construct a target fusion feature that can effectively represent the region of interest. Analyzing the region of interest based on the fused effective features can improve the accuracy of the region of interest classification.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a medical image processing method, apparatus, device and storage medium. Background Art

[0002] Knee osteoarthritis (OA) is one of the most common joint diseases in humans. Approximately 240 million people worldwide suffer from knee OA. If left untreated, knee OA worsens with age and ultimately leads to disability. Therefore, early screening and accurate diagnosis of knee OA are of paramount clinical value. Clinical examination is a key tool for early screening of knee OA. Clinical examination of the knee typically utilizes non-invasive imaging techniques such as X-rays, computed tomography (CT), or magnetic resonance imaging (MRI). MRI, due to its lack of radioactivity and excellent soft tissue imaging, has become the gold standard for diagnosing knee OA. A standard clinical knee examination uses an MRI machine to scan multiple image slices from three viewing angles / orientations: sagittal, coronal, and transverse. Different MRI modalities, including T1, T2, and PD, are used for different viewing angles. Therefore, clinical examination of the knee joint will provide multi-view, multi-modal thick-slice MRI images.

[0003] Currently, clinical examinations based on multi-view, multi-modal MRI images typically use traditional deep learning techniques to classify knee OA clinical MRI images. However, the cartilage in the knee joint is a thin, curved structure attached to the surface of the joint bone, occupying a very small proportion of the MRI image. Using traditional deep learning methods to process this would introduce a large amount of irrelevant background information. Traditional deep learning methods are also mostly targeted at 2D natural images and perform poorly in 3D medical images. These difficulties make it difficult for existing deep learning methods to extract effective features related to knee OA and achieve satisfactory diagnostic accuracy.

[0004] Based on this, how to provide a method for accurately analyzing knee OA based on multi-view and multi-modal MRI images has become an urgent problem to be solved in the current medical field. Summary of the Invention

[0005] The present application provides a medical image processing method, apparatus, device and storage medium, which can analyze knee OA based on multi-view and multi-modal MRI images, thereby improving the grading accuracy of knee OA.

[0006] In a first aspect, the present application provides a medical image processing method. The method comprises:

[0007] Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted;

[0008] Constructing a region of interest map representation based on surface sampling points of the region of interest;

[0009] Extract multiple modal features corresponding to each surface sampling point in the region of interest map representation;

[0010] The multiple modal features corresponding to all surface sampling points in the region of interest graph are fused to obtain the target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0011] In one embodiment, the medical image processing method further includes: acquiring modal images of different perspectives of the medical image to be processed; determining the intersection points between the modal images of different perspectives based on the position information of the modal images of different perspectives in physical space; and extracting surface sampling points of the region of interest using the medical image to be processed, including: performing segmentation processing on the medical image to be processed to obtain a segmented image; and extracting the surface sampling points of the region of interest from the segmented image based on the intersection points.

[0012] In one embodiment, a region of interest graph representation is constructed based on surface sampling points of the region of interest, including: constructing a vertex coordinate matrix corresponding to the region of interest based on coordinate information of each surface sampling point; obtaining image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest; obtaining an adjacency matrix corresponding to the region of interest based on adjacent relationships between surface sampling points; and constructing a region of interest graph representation based on the vertex coordinate matrix, the image block matrix, and the adjacency matrix.

[0013] In one embodiment, segmenting the medical image to be processed to obtain a segmented image includes: segmenting the region of interest on the T1 modality image of the sagittal plane of the medical image to be processed or the T2 modality image of the sagittal plane of the medical image to be processed to obtain a segmented image.

[0014] In one embodiment, image block data of multiple perspectives corresponding to each surface sampling point is obtained to obtain an image block matrix corresponding to the region of interest, including: extracting image block data of different perspectives corresponding to each surface sampling point from modal images of different perspectives of the medical image to be processed with each surface sampling point as the center, to obtain an image block matrix corresponding to the region of interest.

[0015] In one embodiment, extracting multiple modal features corresponding to each surface sampling point in the region of interest image representation includes: performing feature extraction on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest image representation to obtain multiple modal features corresponding to each surface sampling point.

[0016] In one embodiment, multiple modal features corresponding to all surface sampling points in the region of interest map are fused to obtain a target fusion feature of the region of interest, including: fusing multiple modal features corresponding to each surface sampling point to obtain a local fusion feature; fusing the local fusion features corresponding to all surface sampling points to obtain a target fusion feature.

[0017] In a second aspect, the present application further provides a medical image processing device. The device comprises:

[0018] A first extraction module is used to extract surface sampling points of a region of interest from a segmented image of the medical image to be processed;

[0019] A construction module for constructing a region of interest map representation based on surface sampling points of the region of interest;

[0020] A second extraction module is used to extract multiple modal features corresponding to each surface sampling point in the region of interest graph representation;

[0021] The fusion module is used to fuse multiple modal features corresponding to all surface sampling points in the region of interest map representation to obtain the target fusion feature of the region of interest map; the target fusion feature can be used to determine the classification of the region of interest.

[0022] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0023] Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted;

[0024] Constructing a region of interest map representation based on surface sampling points of the region of interest;

[0025] Extract multiple modal features corresponding to each surface sampling point in the region of interest map representation;

[0026] The multiple modal features corresponding to all surface sampling points in the region of interest graph are fused to obtain the target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0028] Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted;

[0029] Constructing a region of interest map representation based on surface sampling points of the region of interest;

[0030] Extract multiple modal features corresponding to each surface sampling point in the region of interest map representation;

[0031] The multiple modal features corresponding to all surface sampling points in the region of interest graph are fused to obtain the target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0032] The present application provides a medical image processing method, apparatus, device and storage medium, which can extract surface sampling points of a region of interest, construct a graphical representation of the region of interest based on the surface sampling points, and then extract multiple modal features of each sampling point in the graphical representation, and fuse the multiple modal features of the surface sampling points, and analyze or grade the region of interest based on the fused features to obtain analysis or grading results. It can be seen that the present application takes into account the correlation of the region of interest under different perspectives and different modalities, fuses the multiple modal features of the surface sampling points, thereby constructing a target fusion feature that can effectively represent the region of interest, and analyzes or grades the region of interest based on the effective features after fusion, thereby improving the accuracy of the analysis or grading of the region of interest. Moreover, the present application analyzes the region of interest based on the effective features, which can avoid the problem of low accuracy of the analysis results due to the introduction of irrelevant background information in the traditional method of analyzing the processed medical image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A diagram showing an application environment of a medical image processing method in one embodiment;

[0034] Figure 2 is a flowchart of a medical image processing method in one embodiment;

[0035] Figure 3 is an MRI image of a knee joint in one embodiment;

[0036] Figure 4 is another flowchart of a medical image processing method according to an embodiment;

[0037] Figure 5 1. Modal images of a knee joint from different perspectives in one embodiment;

[0038] Figure 6 is another flowchart of a medical image processing method according to an embodiment;

[0039] Figure 7 is another flowchart of a medical image processing method according to an embodiment;

[0040] Figure 8 is another flowchart of a medical image processing method according to an embodiment;

[0041] Figure 9 is a structural block diagram of a medical image processing device in one embodiment;

[0042] Figure 10 is another structural block diagram of a medical image processing device in one embodiment;

[0043] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] Knee OA is one of the most common joint diseases in humans. Approximately 240 million people worldwide suffer from knee OA. If not treated promptly, knee OA will worsen as the patient ages and eventually lead to disability. Therefore, early screening and accurate grading of knee OA are of extremely important clinical value. Clinical examination is an important means of early screening for knee OA. Clinical examination of the knee joint generally uses non-invasive imaging techniques such as X-rays, computed tomography (CT) or magnetic resonance imaging (MRI). Among them, MRI has become the gold standard for diagnosing or grading knee OA due to its advantages such as being non-radioactive and having good soft tissue imaging effects. Standard clinical examination of the knee joint uses MRI equipment to scan multiple image slices of the patient from three perspectives: sagittal, coronal and transverse. At the same time, different MRI modalities are used for different perspectives, including T1-weighted, T2-weighted and PD modalities. Therefore, clinical examination of knee OA will provide multi-perspective, multi-modal thick-slice MRI images. It will be understood that the above three sequences are only schematically provided in the embodiments of this application. In other embodiments, the MRI modality may include conventional sequences such as T1-weighted and T2-weighted, or inversion sequences such as fluid inversion recovery (FLAIR) and short T1 inversion recovery (STIR), or a combination of one or more of diffusion weighted imaging (DWI), perfusion weighted imaging (PWI), susceptibility weighted imaging (SWI), diffusion tensor imaging (DTI), and magnetic resonance spectroscopy.

[0046] Alternatively, T1 relaxation can represent the recovery of the longitudinal magnetization (Mz). Mz increases exponentially during relaxation; the higher the Mz when a 90° RF pulse is applied, the larger the transverse signal. T1 relaxation occurs because of energy exchange between rotating nuclei and their surroundings (i.e., the lattice), causing the populations of nuclei in the up and down states to shift, returning to the equilibrium distribution without B1. Consequently, Mz returns to M0, and T1 is also called the spin-lattice relaxation time. Accordingly, T1-weighted imaging (T1WI) can highlight differences in T1 relaxation (longitudinal relaxation) across tissues.

[0047] Alternatively, T2 decay is the decay of the transverse magnetization (Mxy) after a 90° RF pulse. The longer the time after the 90° RF pulse, the greater the Mxy decay and the smaller the transverse signal. Mxy decays exponentially during relaxation, with a time constant of T2. Alternatively, T2 is also called spin-spin relaxation time. Accordingly, T2-weighted imaging (T2WI) can highlight differences in tissue T2 relaxation (transverse relaxation).

[0048] Alternatively, PD / proton density weighted image (PdWI) mainly reflects the difference in proton density relaxation between tissues.

[0049] Currently, clinical examinations based on multi-view, multi-modal MRI images typically use traditional deep learning techniques to classify knee OA clinical MRI. In recent years, deep learning-based computer-aided diagnosis (CAD) technology has seen numerous successful applications in intelligent grading of medical images. Most of these methods are derived from the field of natural imaging. For knee OA grading, deep learning CAD methods can be categorized into two main groups: first, OA grading based on single-view and modality knee MRI images; and second, OA grading based on multi-view, multi-modal knee MRI image fusion.

[0050] OA grading methods based on single-view and modality knee MRI images can be further divided into volume-level, slice-level, and patch-level methods. Volume-level methods use deep learning methods for 3D data, using the entire patient MRI image as input for deep learning network training. However, this method requires small training annotations, which easily introduces a large amount of irrelevant background, resulting in poor network training results. Slice-level methods use a single 2D slice from the 3D MRI image as input. Although this method also introduces a large amount of irrelevant background, it can achieve better training results due to the increased amount of training annotations and the ability to use weights trained from natural images as initial parameters. Patch-level methods rely on joint bone segmentation atlases to sample patches of the knee bone surface and use 2D deep learning methods for training. This method reduces irrelevant background to a certain extent and also achieves good training results. However, the features of knee OA provided by single-view and modality knee MRI images are relatively limited, resulting in low knee OA grading accuracy.

[0051] Among them, the OA grading method based on multi-view multi-modal knee MRI image fusion cannot be directly represented as a multi-channel 3D image or interpolated into a 3D image matrix of the same shape due to the differences in layer thickness, number of layers, field of view (FOV) and other properties of knee MRI images of different viewpoints and modalities. Therefore, the current OA grading methods based on multi-view multi-modal knee MRI image fusion first treat the images of each viewpoint of the knee MRI as an independent case: using a method similar to single-view grading, by separately training the OA grading model for knee MRI images of different viewpoints, giving the feature values ​​of different viewpoints, and then giving the final grading result by fusing the feature values ​​of multiple viewpoints of the same case. However, this method makes each viewpoint independent, and the correlation of the same lesion under different viewpoints is destroyed, making the final fused lesion features inaccurate, which makes it difficult to accurately describe and locate the lesion, and thus makes the grading accuracy of knee OA low.

[0052] Based on this, the present application provides a medical image processing method, apparatus, device and storage medium, which can accurately analyze or grade knee OA based on multi-view and multi-modal MRI images, thereby improving the accuracy and precision of knee OA grading.

[0053] The medical image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The terminal 102 communicates with the scanning device 104 via a network. The data storage system can store the data that the terminal 102 needs to process. The data storage system can be integrated on the terminal 102, or placed on the cloud or other network servers. The scanning device 104 is used to scan a structure containing a certain morphological tissue to obtain a medical image of the morphological tissue structure; the terminal 102 is used to analyze or grade the medical image scanned by the scanning device 104 and obtain an analysis or grading result. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers. The scanning device 104 can be an X-ray device, a CT device, an MRI device, etc.

[0054] In one embodiment, Figure 2 As shown, a medical image processing method is provided, which is applied to Figure 1 The terminal in is taken as an example to illustrate, including the following steps:

[0055] S201: Surface sampling points of a region of interest are extracted using the medical image to be processed.

[0056] The medical image to be processed is a medical image containing a certain morphological tissue structure, for example, the medical image to be processed may be an MRI image of a knee joint; the region of interest is the lesion part of the morphological tissue structure in the medical image to be processed; the surface sampling points of the region of interest are sampling points on the surface of the region of interest in three-dimensional space, for example, they may be sampling points on the surface of the knee joint bone.

[0057] In a specific implementation, the terminal can first obtain the medical image to be processed from an image scanning device (such as an MRI scanning device), then perform image processing (such as segmentation processing) on ​​the medical image to be processed, and extract the image of the region of interest in the medical image to be processed, and then further perform point sampling on the surface of the morphological tissue structure or the lesion part structure contained in the image of the region of interest to obtain the surface sampling points of the region of interest.

[0058] S202: Construct a region of interest map representation based on the surface sampling points of the region of interest.

[0059] The ROI graph representation is a graph data structure that may include the coordinates of all surface sampling points in the ROI, the tile data corresponding to the surface sampling points at different perspectives, and the relationships between adjacent surface sampling points. It should be noted that the ROI graph representation in the embodiments of the present application may also be referred to as ROI graph representation, ROI graphical representation, ROI graphical representation, etc.

[0060] In a specific implementation, when the terminal obtains the surface sampling points of the area of ​​interest, it can further determine the coordinates of each surface sampling point, obtain the tile data of different perspectives corresponding to each surface sampling point, and analyze the relationship between adjacent surface sampling points. Then, based on the coordinates of all surface sampling points in the area of ​​interest, the tile data of different perspectives corresponding to all surface sampling points, and the relationship between adjacent surface sampling points, a map representation of the area of ​​interest is constructed, which is convenient for subsequent feature storage or feature extraction of the area of ​​interest.

[0061] S203: extract multiple modal features corresponding to each surface sampling point in the region of interest image representation.

[0062] Among them, the multiple modal features corresponding to each surface sampling point may include features of three modalities: T1, T2 and PD. In a specific implementation, since the region of interest map representation includes tile data of different perspectives corresponding to all surface sampling points in the region of interest, it is possible to further perform feature extraction or analysis based on the multiple tile data corresponding to each surface sampling point to obtain multiple modal features corresponding to each surface sampling point, for example, obtaining T1 modal features, T2 modal features and PD modal features corresponding to each surface sampling point. Optionally, the terminal may use a convolutional neural network (CNN) to extract multiple modal features of each surface sampling point in the region of interest map representation.

[0063] S204 , fusing multiple modal features corresponding to all surface sampling points in the region of interest graph representation to obtain a target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0064] In a specific implementation, a fusion network or a conversion network (such as the Transformer model in a deep learning model) can be used to fuse multiple modal features corresponding to all surface sampling points in the region of interest graph representation, thereby obtaining the target fusion features of the region of interest.

[0065] In one possible implementation, multiple modal features corresponding to all surface sampling points may be directly fused to obtain target fused features of the region of interest.

[0066] In one possible implementation, multiple modal features corresponding to each surface sampling point can be fused first to obtain the fused features corresponding to each surface sampling point, and then the fused features corresponding to all surface sampling points can be fused to obtain the target fused features of the region of interest.

[0067] After obtaining the target fusion features of the region of interest, the Graph Convolutional Network (GCN) can be used to grade the degree of damage to the region of interest based on the target fusion features to determine the degree of damage to the region of interest. The degree of damage grading of the region of interest can measure the degree of damage, lesion grade, etc. Specifically, this embodiment takes into account that the accuracy required for clinical grading is much lower than the accuracy of the WORMS scale. Therefore, similar classifications in the WORMS scale can be merged to divide the degree of damage into level 0 damage, level 1 damage, and level 2 damage. Among them, level 0 damage indicates that there is no damage to the region of interest, which corresponds to level 0 and level 1 in the WORMS score; level 1 damage indicates mild damage to the region of interest, which corresponds to level 2, level 3, and level 4 in the WORMS score; level 2 damage indicates severe damage to the region of interest, which corresponds to level 5 and level 6 in the WORMS score. For example, Figure 3 As shown, Figure 3 This is an MRI image of the knee joint. Figure 3 a is a schematic diagram of knee cartilage grade 0 injury. Figure 3 b is a schematic diagram of grade 1 knee cartilage injury. Figure 3 c is a schematic diagram of grade 2 knee cartilage injury.

[0068] After obtaining the target fusion features of the region of interest, the Classification Activation Mapping (CAM) can be used to classify the lesion location of the region of interest based on the target fusion features to determine the lesion location classification of the region of interest. The lesion location classification of the region of interest can represent the location of the lesion in the region of interest.

[0069] In a possible implementation, when analyzing or grading the region of interest based on the target fusion feature, only the damage degree of the region of interest may be graded. In this case, the damage degree is the analysis / grading result of the region of interest.

[0070] In a possible implementation, when analyzing or grading the region of interest based on the target fusion feature, only the lesion location of the region of interest may be graded. In this case, the lesion location is the analysis / grading result of the region of interest.

[0071] In one possible implementation, when analyzing or grading the region of interest based on the target fusion features, the damage degree of the region of interest can be graded and the lesion location can be determined at the same time. In this case, the damage degree and lesion location are the analysis / grading results of the region of interest.

[0072] The medical image processing method provided in the embodiment of the present application can extract the surface sampling points of the region of interest, construct a graphical representation of the region of interest based on the surface sampling points, and then extract multiple modal features of each sampling point in the graphical representation, and fuse the multiple modal features of the surface sampling points, and analyze or grade the region of interest based on the fused features to obtain an analysis or grading result. It can be seen that the present application takes into account the correlation of the region of interest under different perspectives and different modalities, fuses the multiple modal features of the surface sampling points, thereby constructing a target fusion feature that can effectively represent the region of interest, and analyzes or grades the region of interest based on the effective features after fusion, thereby improving the accuracy of the analysis or grading of the region of interest. Moreover, the present application analyzes the region of interest based on the effective features, which can avoid the problem of low accuracy of the analysis results due to the introduction of irrelevant background information in the traditional method of analyzing the processed medical image.

[0073] In another embodiment of the present application, a method for extracting surface sampling points of a region of interest using a medical image to be processed is provided, specifically comprising the following steps: Figure 4 Steps shown:

[0074] S401: Acquire modal images of different viewing angles of a medical image to be processed.

[0075] The modal images of different viewing angles of the medical image to be processed include: T1 modal images of the sagittal plane of the medical image to be processed, T2 modal images of the sagittal plane of the medical image to be processed, PD modal images of the coronal plane of the medical image to be processed, and PD modal images of the transverse plane of the medical image to be processed. Figure 5 As shown, Figure 5 Modal images of the knee joint from different perspectives, Figure 5 a is the T1 modality image of the knee joint at the sagittal angle. Figure 5 b is the T2 modality image of the knee joint at the sagittal angle. Figure 5 c is the PD modality image of the knee joint from the coronal perspective, Figure 5 d is the PD modality image of the knee joint from a cross-sectional perspective.

[0076] In a specific implementation, a magnetic resonance imaging device can perform magnetic resonance imaging of morphological tissue structures or lesion structures based on three viewing angles / scanning orientations: sagittal, coronal, and transverse. This allows for the acquisition of four modal MRI images from three viewing angles of the region of interest, representing modal images of the medical image to be processed at different viewing angles. The terminal can connect to the magnetic resonance imaging device and acquire modal images of the medical image to be processed from it at different viewing angles.

[0077] S402: Determine intersection points between modal images at different perspectives based on position information of the modal images at different perspectives in physical space.

[0078] Since modal images of different perspectives are in different fields of view (FOV), the morphological tissue structures in modal images of different perspectives are in different spatial positions. Therefore, when the terminal obtains modal images of different perspectives, it can further perform cross-section processing on the modal images of different perspectives based on the position information of the modal images of each perspective in the physical space to obtain the intersection surface between the modal images of different perspectives, and then obtain the intersection point between the modal images of different perspectives.

[0079] S403 , performing segmentation processing on the medical image to be processed to obtain a segmented image.

[0080] In a specific implementation, the terminal can use any image segmentation algorithm or network to segment the medical image to be processed, thereby segmenting an image of the region of interest from the medical image to be processed. Alternatively, the terminal can use a U-Net network to perform semantic segmentation on the medical image to remove irrelevant background from the medical image to extract the region of interest, thereby obtaining a segmented image of the medical image to be processed.

[0081] It should be noted that the medical image to be processed for segmentation processing is the T1 modal image of the sagittal plane of the medical image to be processed or the T2 modal image of the sagittal plane of the medical image to be processed, that is, the T1 modal image of the sagittal plane of the medical image to be processed or the T2 modal image of the sagittal plane of the medical image to be processed can be subjected to segmentation processing of the region of interest to obtain a segmented image.

[0082] S404 , extracting surface sampling points of the region of interest from the segmented image according to the intersection points.

[0083] In a specific implementation, the terminal can sample the intersection points along the surface of the region of interest in the segmented image, thereby eliminating background areas that are not related to the region of interest; optionally, the terminal can also extract the intersection points on the surface of the region of interest in the physical space based on the semantic information of the segmented image, which are the surface sampling points of the region of interest.

[0084] An embodiment of the present application provides a method for extracting surface sampling points of a region of interest based on modal images of different perspectives of a medical image to be processed. Specifically, modal images of four modalities of three perspectives of the medical image to be processed are first obtained, and the intersection points between the modal images of different perspectives are determined based on the position information of the modal images of different perspectives in the physical space. The medical image to be processed is then segmented to segment the region of interest in the medical image to be processed, and finally, the intersection points on the surface of the region of interest located in the physical space are extracted from the intersection points, which are the surface sampling points of the region of interest. It can be seen that the present application can extract the surface sampling points of the region of interest from the intersection points between the modal images of different perspectives of the medical image to be processed based on the segmented image containing only the region of interest, thereby avoiding the introduction of irrelevant information and ensuring the accuracy of the surface sampling points of the collected region of interest. Furthermore, the region of interest is analyzed or graded based on accurate surface sampling points, thereby improving the accuracy of the grading of the region of interest.

[0085] The embodiment described above introduces a scheme for constructing a region of interest graph representation based on the surface sampling points of the region of interest. In another embodiment of the present application, a region of interest graph representation can be constructed based on the vertex coordinate matrix, image block matrix and adjacency matrix of the surface sampling points of the region of interest. For example, the “constructing a region of interest graph representation based on the surface sampling points of the region of interest” mentioned above specifically includes the following: Figure 6 Steps shown:

[0086] S601: Construct a vertex coordinate matrix corresponding to the region of interest according to the coordinate information of each surface sampling point.

[0087] In the specific implementation, the coordinate information of each surface sampling point is collected, all the coordinate information is combined into a matrix, and a matrix containing the coordinates of all surface sampling points in the physical space is constructed, which is the vertex coordinate matrix corresponding to the area of ​​interest.

[0088] S602 : Acquire image block data of multiple viewing angles corresponding to each surface sampling point, and obtain an image block matrix corresponding to the region of interest.

[0089] In a specific implementation, based on the modal images of different perspectives of the medical image to be processed, the image blocks containing any surface sampling point in the modal images of different perspectives are extracted, which are the image block data of multiple perspectives corresponding to the surface sampling point. The image block data of all surface sampling points are grouped into a matrix, which is the image block matrix corresponding to the region of interest. For example, a surface sampling point corresponds to the image block data of the sagittal plane T1, the image block data of the sagittal plane T2, the image block data of the coronal plane PD, and the image block data of the transverse plane PD. Among them, the image block data of multiple perspectives corresponding to each surface sampling point can be a 2*2 image block, a 3*3 image block, or a 4*4 image block, and this application does not impose any restrictions on this.

[0090] In one possible implementation, image block data corresponding to each surface sampling point at different viewpoints can be extracted from the modal images of the medical image to be processed at different viewpoints, with each surface sampling point as the center, to obtain an image block matrix corresponding to the region of interest. In this case, when the surface sampling point is located at the center of the image block, the image block data corresponding to each surface sampling point at multiple viewpoints is an odd-numbered image block, such as a 3*3 image block, a 5*5 image block, or a 7*7 image block.

[0091] S603: Obtain an adjacency matrix corresponding to the region of interest based on the adjacent relationship between the surface sampling points.

[0092] The adjacency matrix corresponding to each surface sampling point can represent the relationship between each surface sampling point and its adjacent surface sampling points; the adjacency matrix corresponding to the region of interest can represent the relationship between all adjacent surface sampling points.

[0093] In a specific implementation, multiple adjacent surface sampling points of any surface sampling point can be determined first, and then an adjacency matrix corresponding to the surface sampling point and its corresponding adjacent surface sampling points can be constructed. The adjacency matrices of all surface sampling points can be combined into one matrix to obtain the adjacency matrix corresponding to the region of interest.

[0094] S604: Construct a region of interest graph representation according to the vertex coordinate matrix, the image block matrix, and the adjacency matrix.

[0095] In a specific implementation, the terminal may combine the vertex coordinate matrix, image block matrix and adjacency matrix of the region of interest into a data set, that is, form a graph data structure, thereby obtaining a graph representation of the region of interest.

[0096] The embodiment of the present application provides a method for constructing a region of interest graph representation based on the vertex coordinate matrix, image block matrix and adjacency matrix of the region of interest. Specifically, the vertex coordinate matrix of the region of interest is constructed based on the coordinates of all surface sampling points, the image block data of the region of interest is constructed based on the image block data of multiple perspectives corresponding to all surface sampling points, the adjacency matrix of the region of interest is constructed based on the relationship between adjacent surface sampling points, and the region of interest graph representation containing all the above matrices is constructed based on the vertex coordinate matrix, image block matrix and adjacency matrix of the region of interest. It can be seen that the region of interest graph representation contains data such as the position of each surface sampling point, adjacent surface sampling points and the relationship between adjacent surface sampling points, so that the multiple modal features of each surface sampling point extracted based on the region of interest graph representation are more accurate. Furthermore, the region of interest is analyzed or graded based on accurate modal features, and the analysis or grading results obtained are also more accurate.

[0097] The above-mentioned embodiment introduces a scheme for extracting multiple modal features corresponding to each surface sampling point. In another embodiment of the present application, multiple modal features can be extracted based on the image block data of each surface sampling point. For example, the above-mentioned "extracting multiple modal features corresponding to each surface sampling point in the region of interest image representation" specifically includes:

[0098] Feature extraction is performed on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest image representation to obtain multiple modal features corresponding to each surface sampling point.

[0099] In a specific implementation, the image block data of each surface sampling point in the image block data of the region of interest in the region of interest map representation can be first extracted, and then a feature extraction network (for example, a CNN network) is used to perform feature extraction on the image block data of multiple perspectives corresponding to each surface sampling point, thereby obtaining multiple modal features corresponding to each surface sampling point.

[0100] The present embodiment provides a method for extracting multiple modal features based on image block data for each surface sampling point. Specifically, the method extracts multiple modal features corresponding to each surface sampling point from image block data from multiple viewpoints corresponding to each surface sampling point. This method allows the extraction of different modal features for each surface sampling point based on a constructed region of interest map representation, making the entire modal feature extraction process simple and convenient.

[0101] The embodiment described above introduces a scheme for fusing multiple modal features of all surface sampling points to obtain target fusion features of the region of interest. In another embodiment of the present application, multiple modal features of all surface sampling points can be locally fused and globally fused in sequence to obtain target fusion features of the region of interest. For example, the above-mentioned "fusing multiple modal features corresponding to all surface sampling points in the region of interest map representation to obtain target fusion features of the region of interest" specifically includes the following: Figure 7 Steps shown:

[0102] S701: Fuse multiple modal features corresponding to each surface sampling point to obtain a local fusion feature.

[0103] Among them, the local fusion feature is the feature obtained by fusion of the features of images of different viewpoints and modalities corresponding to each surface sampling point.

[0104] In a specific implementation, when the terminal fuses multiple modal features corresponding to each surface sampling point, it can first convert the modal features of each viewpoint corresponding to each surface sampling point into a one-dimensional vector. Then, a fusion network is used to fuse the one-dimensional vectors corresponding to each surface sampling point to obtain a fused one-dimensional vector. That is, the fused one-dimensional vector can represent a local fusion feature. This embodiment fuses multiple modal features corresponding to each surface sampling point to achieve local feature fusion of the region of interest. This local fusion feature can effectively and accurately reflect the information of the region of interest, thereby improving the accuracy of subsequent analysis based on the local fusion feature.

[0105] S702: Fuse the local fusion features corresponding to all surface sampling points to obtain target fusion features.

[0106] In a specific implementation, the terminal can first use a fusion network or a conversion network (such as a Transformer network) to fuse multiple modal features corresponding to each surface sampling point to obtain a fusion feature corresponding to each surface sampling point, which is a local fusion feature. Then, the fusion network or the conversion network (such as a Transformer network) is still used to fuse the local fusion features corresponding to all surface sampling points to obtain a fusion feature corresponding to the region of interest, which is a target fusion feature. It should be noted that when the terminal fuses the local fusion features corresponding to all surface sampling points, the vertex coordinate matrix, adjacency matrix and image block matrix corresponding to the region of interest graph representation can be input into the fusion network or the conversion network for feature fusion to obtain the target fusion feature, so that the target fusion feature can reflect the correlation between the surface sampling points, thereby improving the accuracy of the subsequent analysis based on the target fusion feature. In addition, when the terminal fuses the local fusion features corresponding to all surface sampling points, it can specifically perform graph convolution calculation on the local fusion features corresponding to all surface sampling points, and finally obtain the target fusion feature through the global pooling layer.

[0107] The embodiment of the present application provides a method for fusing multiple modal features of all surface sampling points twice to obtain a target fusion feature. Specifically, the multiple modal features corresponding to each surface sampling point are first fused to obtain a local fusion feature corresponding to each surface sampling point, and then the local fusion features corresponding to all surface sampling points are fused to obtain a target fusion feature corresponding to the region of interest. It can be seen that the embodiment of the present application takes into account the correlation of the region of interest under different perspectives and different modalities, and fuses the multiple modal features of the surface sampling points, thereby constructing a target fusion feature that can effectively represent the region of interest. Furthermore, the region of interest is analyzed or graded based on the effective feature, thereby improving the accuracy of the analysis or grading of the region of interest. Moreover, the present application analyzes the region of interest based on the effective feature, which can avoid the problem of low accuracy of the analysis result due to the introduction of irrelevant background information in the traditional method of analyzing the processed medical image.

[0108] refer to Figure 8 , Figure 8 Another flowchart of the medical image processing method provided in the embodiment of the present application. Taking the region of interest as the knee joint as an example, the medical image processing method provided in the embodiment of the present application may include the following steps:

[0109] S801. Acquire multi-view and multi-modal MRI images of the knee joint, wherein the multi-view and multi-modal images include sagittal T1, sagittal T2, coronal PD, and transverse PD.

[0110] S802. Use the U-Net network to segment the knee joint atlas from the sagittal T2 knee joint MRI image.

[0111] S803. Map the multi-view and multi-modal knee joint MRI images into physical space to obtain the field of view (FOV) of each view.

[0112] S804: Determine the intersection plane between the multi-view and multi-modal knee joint MRI images based on the FOV of each view.

[0113] S805 . Determine the intersection points between the multi-view and multi-modal knee joint MRI images based on the intersection planes between the multi-view and multi-modal knee joint MRI images.

[0114] S806. Based on the knee joint atlas, extract knee joint vertices (ie, surface sampling points of the knee joint) from intersections between multi-view and multi-modal knee joint MRI images.

[0115] S807. Construct a knee joint graph representation based on all knee joint vertices, which includes multi-view blocks (i.e., multi-view image block data), vertex coordinates, and an adjacency matrix.

[0116] S808 , performing feature extraction on the multi-view image block of each vertex respectively to obtain multiple modal features corresponding to each vertex.

[0117] S809: Use the Transformer network to fuse multiple modal features corresponding to each vertex to obtain local fusion features.

[0118] S810. Fusing the local fusion features corresponding to all vertices according to the vertex coordinates and the adjacency matrix in the knee joint graph representation to obtain a target fusion feature corresponding to the knee joint.

[0119] S811. Analyze or grade the knee joint based on the target fusion features to obtain an analysis or grading result.

[0120] S812. Determine the grade corresponding to the degree of knee OA injury or the location of the knee joint based on the analysis or grading results of the knee joint.

[0121] It should be noted that Figure 8 The steps in the figure correspond to the steps in the above-mentioned embodiment, and the figure only briefly shows the main meaning of each step. Figure 8 V0, V1, V2, and V3 represent each knee joint vertex in the knee joint graph. Figure 8 The lines between V0, V1, V2, and V3 represent the adjacency relationship between adjacent knee joint vertices in the knee joint graph representation. Figure 8The schematic diagram composed of V0, V1, V2, and V3 represents the knee joint diagram. The diagram only shows four knee joint vertices for illustration, and does not limit the number of knee joint vertices.

[0122] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0123] Based on the same inventive concept, embodiments of the present application also provide a medical image processing device for implementing the aforementioned medical image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more medical image processing device embodiments provided below can be found in the above-described limitations of the medical image processing method and will not be further elaborated here.

[0124] In one embodiment, Figure 9 As shown, a medical image processing device is provided, comprising: a first extraction module, a construction module, a second extraction module and a fusion module, wherein:

[0125] A first extraction module 901 is used to extract surface sampling points of a region of interest from a segmented image of a medical image to be processed;

[0126] A construction module 902 is used to construct a region of interest map representation based on surface sampling points of the region of interest;

[0127] A second extraction module 903 is used to extract multiple modal features corresponding to each surface sampling point in the region of interest graph representation;

[0128] The fusion module 904 is used to fuse multiple modal features corresponding to all surface sampling points in the region of interest map representation to obtain a target fusion feature of the region of interest map; the target fusion feature can be used to determine the classification of the region of interest.

[0129] In one embodiment, based on the above embodiment, Figure 10As shown, the medical image processing apparatus further includes a determination module 905: acquiring modal images of different perspectives of the medical image to be processed; determining intersection points between the modal images of different perspectives based on position information of the modal images of different perspectives in physical space; and extracting surface sampling points of the region of interest using the medical image to be processed, including: performing segmentation processing on the medical image to be processed to obtain a segmented image; and extracting surface sampling points of the region of interest from the segmented image based on the intersection points.

[0130] In one embodiment, based on the above embodiment, construction module 902 is specifically used to construct a vertex coordinate matrix corresponding to the region of interest based on the coordinate information of each surface sampling point; obtain image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest; obtain an adjacency matrix corresponding to the region of interest based on the adjacent relationship between the surface sampling points; and construct a graph representation of the region of interest based on the vertex coordinate matrix, the image block matrix, and the adjacency matrix.

[0131] In one embodiment, based on the above embodiment, the T1 modality image of the sagittal plane of the medical image to be processed or the T2 modality image of the sagittal plane of the medical image to be processed is segmented to obtain a segmented image.

[0132] In one embodiment, based on the above embodiment, with each surface sampling point as the center, image block data of different perspectives corresponding to each surface sampling point is extracted from modal images of different perspectives of the medical image to be processed to obtain an image block matrix corresponding to the region of interest.

[0133] In one embodiment, based on the above embodiment, feature extraction is performed on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest map representation to obtain multiple modal features corresponding to each surface sampling point.

[0134] In one embodiment, based on the above embodiment, the fusion module 904 is specifically used to fuse multiple modal features corresponding to each surface sampling point to obtain a local fusion feature; and fuse the local fusion features corresponding to all surface sampling points to obtain a target fusion feature.

[0135] Each module in the aforementioned medical image processing apparatus may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0136] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external computer device in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a medical image processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0137] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0139] Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted;

[0140] Constructing a region of interest map representation based on surface sampling points of the region of interest;

[0141] Extract multiple modal features corresponding to each surface sampling point in the region of interest map representation;

[0142] The multiple modal features corresponding to all surface sampling points in the region of interest graph are fused to obtain the target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0143] In one embodiment, when the processor executes the computer program, it further implements the following steps: acquiring modal images of different perspectives of the medical image to be processed; determining the intersection points between the modal images of different perspectives based on the position information of the modal images of different perspectives in physical space; and extracting surface sampling points of the region of interest using the medical image to be processed, including: performing segmentation processing on the medical image to be processed to obtain a segmented image; and extracting the surface sampling points of the region of interest from the segmented image based on the intersection points.

[0144] In one embodiment, when executing the computer program, the processor further implements the following steps: constructing a vertex coordinate matrix corresponding to the region of interest based on the coordinate information of each surface sampling point; obtaining image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest; obtaining an adjacency matrix corresponding to the region of interest based on the adjacent relationship between the surface sampling points; and constructing a graph representation of the region of interest based on the vertex coordinate matrix, the image block matrix, and the adjacency matrix.

[0145] In one embodiment, when the processor executes the computer program, it further implements the following steps: performing region of interest segmentation processing on the T1 modality image of the sagittal plane of the medical image to be processed or the T2 modality image of the sagittal plane of the medical image to be processed to obtain a segmented image.

[0146] In one embodiment, when the processor executes the computer program, the following steps are further implemented: with each surface sampling point as the center, image block data of different perspectives corresponding to each surface sampling point is extracted from modal images of different perspectives of the medical image to be processed to obtain an image block matrix corresponding to the region of interest.

[0147] In one embodiment, when the processor executes the computer program, it further implements the following steps: performing feature extraction on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest map representation to obtain multiple modal features corresponding to each surface sampling point.

[0148] In one embodiment, when the processor executes the computer program, it further implements the following steps: fusing multiple modal features corresponding to each surface sampling point to obtain a local fusion feature; fusing the local fusion features corresponding to all surface sampling points to obtain a target fusion feature.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0150] Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted;

[0151] Constructing a region of interest map representation based on surface sampling points of the region of interest;

[0152] Extract multiple modal features corresponding to each surface sampling point in the region of interest map representation;

[0153] The multiple modal features corresponding to all surface sampling points in the region of interest graph are fused to obtain the target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0154] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: acquiring modal images of different perspectives of the medical image to be processed; determining the intersection points between the modal images of different perspectives based on the position information of the modal images of different perspectives in physical space; and extracting surface sampling points of the region of interest using the medical image to be processed, including: performing segmentation processing on the medical image to be processed to obtain a segmented image; and extracting the surface sampling points of the region of interest from the segmented image based on the intersection points.

[0155] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: constructing a vertex coordinate matrix corresponding to the region of interest based on the coordinate information of each surface sampling point; obtaining image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest; obtaining an adjacency matrix corresponding to the region of interest based on the adjacent relationship between the surface sampling points; and constructing a graph representation of the region of interest based on the vertex coordinate matrix, the image block matrix, and the adjacency matrix.

[0156] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: performing region of interest segmentation processing on the T1 modality image of the sagittal plane of the medical image to be processed or the T2 modality image of the sagittal plane of the medical image to be processed to obtain a segmented image.

[0157] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: with each surface sampling point as the center, image block data of different perspectives corresponding to each surface sampling point is extracted from modal images of different perspectives of the medical image to be processed to obtain an image block matrix corresponding to the region of interest.

[0158] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: feature extraction is performed on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest map representation to obtain multiple modal features corresponding to each surface sampling point.

[0159] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: fusing multiple modal features corresponding to each surface sampling point to obtain a local fusion feature; and fusing the local fusion features corresponding to all surface sampling points to obtain a target fusion feature.

[0160] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0161] Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted;

[0162] Constructing a region of interest map representation based on surface sampling points of the region of interest;

[0163] Extract multiple modal features corresponding to each surface sampling point in the region of interest map representation;

[0164] The multiple modal features corresponding to all surface sampling points in the region of interest graph are fused to obtain the target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest.

[0165] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: acquiring modal images of different perspectives of the medical image to be processed; determining the intersection points between the modal images of different perspectives based on the position information of the modal images of different perspectives in physical space; and extracting surface sampling points of the region of interest using the medical image to be processed, including: performing segmentation processing on the medical image to be processed to obtain a segmented image; and extracting the surface sampling points of the region of interest from the segmented image based on the intersection points.

[0166] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: constructing a vertex coordinate matrix corresponding to the region of interest based on the coordinate information of each surface sampling point; obtaining image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest; obtaining an adjacency matrix corresponding to the region of interest based on the adjacent relationship between the surface sampling points; and constructing a graph representation of the region of interest based on the vertex coordinate matrix, the image block matrix, and the adjacency matrix.

[0167] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: performing region of interest segmentation processing on the T1 modality image of the sagittal plane of the medical image to be processed or the T2 modality image of the sagittal plane of the medical image to be processed to obtain a segmented image.

[0168] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: with each surface sampling point as the center, image block data of different perspectives corresponding to each surface sampling point is extracted from modal images of different perspectives of the medical image to be processed to obtain an image block matrix corresponding to the region of interest.

[0169] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: feature extraction is performed on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest map representation to obtain multiple modal features corresponding to each surface sampling point.

[0170] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: fusing multiple modal features corresponding to each surface sampling point to obtain a local fusion feature; and fusing the local fusion features corresponding to all surface sampling points to obtain a target fusion feature.

[0171] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0172] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A medical image processing method, characterized in that: The method comprises: Acquiring modal images of different viewing angles of a medical image to be processed, and determining intersections between the modal images of different viewing angles based on position information of the modal images of different viewing angles in physical space; the medical image to be processed is a three-dimensional medical image; Utilizing the medical image to be processed, surface sampling points of the region of interest are extracted; constructing a region of interest map representation based on surface sampling points of the region of interest; Extracting a plurality of modal features corresponding to each surface sampling point in the region of interest graphical representation; fusing multiple modal features corresponding to all surface sampling points in the region of interest image representation to obtain a target fused feature of the region of interest; the target fused feature can be used to determine the classification of the region of interest; The step of extracting surface sampling points of the region of interest using the medical image to be processed includes: The medical image to be processed is segmented to obtain a segmented image; and surface sampling points of a region of interest are extracted from the segmented image according to the intersection points.

2. The method according to claim 1, characterized in that The constructing a region of interest map representation according to the surface sampling points of the region of interest comprises: Constructing a vertex coordinate matrix corresponding to the region of interest according to the coordinate information of each surface sampling point; Acquire image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest; Obtaining an adjacency matrix corresponding to the region of interest based on the adjacent relationship between the surface sampling points; The region of interest graph representation is constructed according to the vertex coordinate matrix, the image block matrix and the adjacency matrix.

3. The method according to claim 1, characterized in that The step of segmenting the medical image to be processed to obtain a segmented image includes: The region of interest is segmented on the T1 modality image of the sagittal plane of the medical image to be processed or the T2 modality image of the sagittal plane of the medical image to be processed to obtain the segmented image.

4. The method according to claim 2, characterized in that The acquiring of image block data of multiple perspectives corresponding to each surface sampling point to obtain an image block matrix corresponding to the region of interest includes: Taking each of the surface sampling points as the center, image block data of different perspectives corresponding to each of the surface sampling points are extracted from the modal images of different perspectives of the medical image to be processed to obtain an image block matrix corresponding to the region of interest.

5. The method according to claim 2, characterized in that The extracting of multiple modal features corresponding to each surface sampling point in the region of interest image representation includes: Feature extraction is performed on image block data of multiple perspectives corresponding to each surface sampling point in the region of interest image representation to obtain multiple modal features corresponding to each surface sampling point.

6. The method according to claim 1, characterized in that The fusing of multiple modal features corresponding to all surface sampling points in the region of interest graph representation to obtain target fused features of the region of interest includes: Fusing the multiple modal features corresponding to each surface sampling point to obtain a local fusion feature; The local fusion features corresponding to all surface sampling points are fused to obtain the target fusion features.

7. A medical image processing device, characterized in that: The device comprises: a determination module, configured to obtain modal images of different viewing angles of a medical image to be processed, and determine intersection points between the modal images of different viewing angles based on position information of the modal images of different viewing angles in physical space; the medical image to be processed is a three-dimensional medical image; A first extraction module is used to extract surface sampling points of a region of interest from a segmented image of the medical image to be processed; A construction module, configured to construct a region of interest map representation based on surface sampling points of the region of interest; A second extraction module is used to extract multiple modal features corresponding to each surface sampling point in the region of interest image representation; a fusion module, configured to fuse multiple modal features corresponding to all surface sampling points in the region of interest image representation to obtain a target fusion feature of the region of interest; the target fusion feature can be used to determine the classification of the region of interest; The first extraction module is specifically configured to segment the medical image to be processed to obtain a segmented image; and extract surface sampling points of an area of ​​interest from the segmented image according to the intersection points.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images