A computer-aided detection method and system based on spherical convolution

Abstract

The present invention discloses a computer-aided detection method and system based on spherical convolution, which relates to the field of artificial intelligence technology, and comprises: receiving multimodal images, preprocessing the multimodal images to obtain preprocessed multimodal images, mapping the preprocessed multimodal images to a curved surface, expanding the mapped curved surface into a sphere using area-preserving deformation, and generating a spherical convolution kernel using the sphere; encoding other spherical vertices within the spherical vertex field of the spherical convolution kernel and performing convolution operations to obtain all spherical vertices; clustering all spherical vertices into clusters, and performing spherical pooling and de-pooling operations within the clusters; receiving tissue surface information, inputting the tissue surface information into a pre-established segmentation and classification spherical convolution neural network model, and outputting an auxiliary detection result, thereby achieving the representation of tissue and lesion surface image information, and thus assisting the computer in detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a computer-aided detection method and system based on spherical convolution. Background Art

[0002] Imaging technology can non-invasively visualize in vivo tissue structure and lesions, including computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and diffusion weighted imaging (DWI). Current computer-aided detection methods for imaging mainly rely on imaging omics and convolutional neural network (CNN) technologies to characterize and learn the image information of tissue structure and lesion area, and then establish a classification prediction model. Among them, imaging omics methods extract imaging omics features from the region of interest (ROI) with high throughput, including first-order statistical features, shape features, texture features, and wavelet features, and then perform feature screening and establish a machine learning classification model. The CNN model automatically extracts the deep features of the ROI by stacking multiple convolutional and pooling layers, and then performs classification prediction through a fully connected layer. However, imaging omics and CNN methods are usually used to extract features of bulk tissue and lesion ROIs, and their ability to characterize tissue surface information is limited, which restricts the development of computer-aided detection systems in related fields. Summary of the invention

[0003] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a computer-aided detection method and system based on spherical convolution, which realizes the characterization of tissue and lesion surface image information, thereby assisting the computer in detection.

[0004] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a computer-aided detection method based on spherical convolution, the method comprising the following steps:

[0005] Receiving a multimodal image, preprocessing the multimodal image to obtain a preprocessed multimodal image, mapping the preprocessed multimodal image to a surface, and expanding the mapped surface into a spherical surface using an area-preserving deformation;

[0006] Encode other spherical vertices within the spherical vertex field of the spherical convolution kernel to establish a spherical convolution kernel, and apply spherical convolution to all spherical vertices; cluster all spherical vertices into clusters, and perform spherical pooling and unpooling operations within the clusters;

[0007] Receive tissue surface information, input the tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and output an auxiliary detection result, wherein the pre-established segmentation and classification spherical convolutional neural network model is established based on spherical convolution, spherical pooling and de-pooling.

[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of preprocessing the multimodal image:

[0009] Structural MRI preprocessing steps include head motion correction, removal of non-brain tissue, pixel value normalization, spatial standardization, gray-white matter segmentation, gray-white matter interface surface mesh reconstruction, and topology correction;

[0010] Functional MRI preprocessing steps, removal of the first 10 time points, temporal layer correction, head motion correction, registration, smoothing, removal of linear drift, covariate regression, and bandpass filtering;

[0011] Diffusion MRI preprocessing steps, eddy current correction, registration, fitting of water molecule diffusion model and fiber bundle tracking;

[0012] PET preprocessing steps, partial volume effect correction; pixel value normalization;

[0013] Region of interest segmentation based on UNet and ViT;

[0014] The segmentation result is anti-aliased using a sparse field level set, and then the surface is reconstructed using the marching cubes algorithm.

[0015] In combination with the first aspect, in some implementations of the first aspect, the method further includes: mapping the preprocessed multimodal image to a curved surface, and expanding the mapped curved surface into a spherical surface using an area-preserving deformation:

[0016] Mapping the tissue surface information within the neighborhood of the surface vertices to the cortex using a sphere-based mapping, a cylinder-based mapping, or a Gaussian convolution-based mapping method;

[0017] Use area-preserving deformation to expand the surface into a sphere, keeping the area of ​​the surface triangles unchanged during the expansion process;

[0018] Scale the sphere to the unit sphere and resample the vertices.

[0019] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the sphere-based mapping includes:

[0020] For any vertex p on the surface, create a sphere with radius R with p as the center;

[0021] Intersect the sphere with the region of interest, and divide the intersection into one or more connected regions based on pixel connectivity;

[0022] Take the connected region containing the pixel with the closest Euclidean distance to vertex p;

[0023] The pixel information in the connected area is counted and stored in the vertex p.

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the cylinder-based mapping step includes:

[0025] For any vertex p on the surface, establish a cylinder with base radius r and height h, taking p as the base center and the surface normal vector as the axis;

[0026] Intersect the cylinder with the region of interest and divide the intersection into one or more connected regions based on pixel connectivity;

[0027] Take the connected region containing the pixel with the closest Euclidean distance to vertex p;

[0028] The pixel information in the connected area is counted and stored in the vertex p.

[0029] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the Gaussian convolution-based mapping step includes:

[0030] For a vertex p on the surface, let the unit normal vector pointing to the inside of the surface be n. Given the covariance matrix of the multidimensional Gaussian distribution,

[0031]

[0032] in represents the variance along the normal vector direction, Represents the variance perpendicular to the normal vector direction;

[0033] Search for the pixel point in the area of ​​interest closest to vertex p, mark it as visited, and add the vertex to queue Q;

[0034] Assume the unit normal vector of vertex p is n, pop up the head pixel u of the Q queue, and calculate the following formula:

[0035]

[0036] Where x is a vector starting from p and ending at u, 2 is the chi-square distribution, is the preset value, L2 means taking the L2 norm of the vector within the double vertical line ‖·‖, that is, taking the square root of the sum of the squares of the vector elements.

[0037] If d≤1, record the pixel and its density value Mark the unvisited vertices adjacent to u as visited and add them to queue Q in sequence, ensuring that the sampled pixels are all within the 95% variance ellipsoid of vertex p;

[0038] Repeat the above steps until the queue Q is empty;

[0039] Normalize the recorded probability density;

[0040] The pixel expectations in the neighborhood of vertex p are counted based on the normalized probability density and stored at vertex p.

[0041] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the step of encoding other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish the spherical convolution kernel is as follows:

[0042] For the vertex p on the unit sphere, take the great circle between it and the North Pole of the sphere as the reference line, and take the great circles whose tangents are at angles of 45°, 90° and 135° with the tangents of the reference line;

[0043] Divide the sphere into 8 equal quadrants, then take a set of small circles with a geodesic radius of k×δ with vertex p, where k is a positive integer and k≤K, δ is a floating point number, and δ is initialized to the mean of the length of the edge of the spherical surface plus twice the standard deviation;

[0044] The small circle set and the hexagram large circle divide the neighborhood of vertex p into 8K sectors, and assign weight coefficients w0, w1, ..., w to be learned to vertex p and sector rings. 8K ;

[0045] Let the eigenvector of vertex p be h0, and let the fan ring C i There is n i vertices, among which vertex p j The eigenvector of i，j , the spherical convolution takes the mean of the feature vectors in the same fan ring and aggregates them to the vertex p:

[0046]

[0047] Where h is the feature vector of the aggregated vertex p, w0 is the coefficient to be trained for the current vertex p, and w i is the i-th sector ring C iThe coefficient to be trained;

[0048] The spherical convolution kernel is the expansion of the image convolution kernel on the sphere. The fan ring is analogous to the pixel, and K and δ are analogous to the size of the image convolution kernel and the pixel size, respectively.

[0049] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of performing spherical pooling and de-pooling operations within the cluster:

[0050] Use the KMeans++ method to randomly select M vertices on the sphere. First, randomly select the first center point and calculate its geodesic distance to other vertices in the area of ​​interest, which is the length of the minor arc of the great circle of the sphere.

[0051] Establish a probability distribution proportional to the square of the geodesic distance, randomly select a new center point and update the geodesic distance from the vertex to the center point until M center points are selected;

[0052] Taking the selected M vertices as the initial centers, K-center point clustering is used to assign vertices to M clusters according to the minimum geodesic distance to the center point;

[0053] Select the vertex with the smallest sum of geodesic distances to other vertices in the cluster as the new center point, and continue iterating until convergence;

[0054] Spherical pooling gathers vertex information within a cluster to the center point, and average pooling and maximum pooling are optional;

[0055] Spherical anti-pooling redistributes the center point information to other vertices in the cluster, and selects anti-average pooling and anti-maximum pooling.

[0056] In combination with the first aspect, in some implementations of the first aspect, the method further includes: inputting tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and outputting a process of obtaining an auxiliary detection result:

[0057] Perform multiple spherical convolution and spherical pooling operations on the spherical surface;

[0058] Input the feature vector of the spherical vertices into the fully connected layer;

[0059] Use spherical unpooling and spherical convolution to reconstruct the spherical surface so that the input and output are the same;

[0060] Extract the bottleneck layer features of the fully connected layer and build a machine learning model as a spherical convolutional neural network model for segmentation and classification;

[0061] The encoder is constructed using multiple layers of spherical convolutional layers and spherical pooling layers to extract deep features of the tissue surface.

[0062] The spherical convolution layer can use cross-layer connections;

[0063] When the output is classification or grading, the encoder output is passed to the fully connected layer, and the fully connected layer outputs the classification or decomposition result;

[0064] When the output is semantic segmentation, a decoder is constructed using multiple layers of spherical convolutional layers and spherical unpooling layers. The decoder reconstructs the same sphere as the input, with the vertices being the labels of the semantic segmentation.

[0065] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a computer-aided detection system based on spherical convolution, comprising:

[0066] A spherical surface generation module is used to receive a multimodal image, preprocess the multimodal image to obtain a preprocessed multimodal image, map the preprocessed multimodal image to a curved surface, and expand the mapped curved surface into a spherical surface using an area-preserving deformation;

[0067] The pooling and unpooling modules are used to encode other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish the spherical convolution kernel, apply the spherical convolution to all spherical vertices; cluster all spherical vertices into clusters, and perform spherical pooling and unpooling operations within the clusters;

[0068] The auxiliary detection module is used to receive tissue surface information, input the tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and output an auxiliary detection result, wherein the pre-established segmentation and classification spherical convolutional neural network model is established based on spherical convolution, spherical pooling and de-pooling.

[0069] Beneficial effects of the present invention:

[0070] The surface mapping method of tissue surface information of the present invention projects the surface information of the area of ​​interest in the image onto the grid surface, and has the advantages of parallel operation and strong robustness; the grid surface of the tissue is deformed into a sphere, and a spherical convolution method is established on the spherical surface. The method generalizes the convolution operation from the image to the spherical grid surface, which solves the limitations of the traditional convolution method in highly complex physiological tissue surfaces, and has the advantages of a small number of parameters and easy calculation. A spherical convolution neural network is established based on the spherical convolution and spherical pooling methods for computer-aided detection to improve detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0072] Figure 1 It is a schematic flow chart of the method of the present invention;

[0073] Figure 2 is a schematic diagram of encoding spherical vertices by a spherical convolution operation in an embodiment of the present invention;

[0074] Figure 3 is a structural block diagram of spherical pooling in an embodiment of the present invention;

[0075] Figure 4 It is a schematic diagram of the system structure of the present invention;

[0076] Figure 5 Schematic diagram of a spherical convolutional residual network for brain spherical mapping and classification in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Embodiment 1:

[0079] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0080] Spherical convolution: The basic principle of spherical convolution is to define the convolution operation on a sphere. Similar to convolution on a two-dimensional plane, spherical convolution is also implemented by sliding a filter in a neighborhood around each pixel. However, due to the special properties of the sphere, the curvature and projection effects on the sphere must be considered.

[0081] Convolution on a sphere can be measured by the distance between points on the sphere. The commonly used measurement method is the geodesic distance on the sphere, which is the shortest arc distance between two points. In the convolution operation, the size and shape of the filter can be selected as needed, and common shapes include circular and elliptical.

[0082] like Figure 1 As shown, a computer-aided detection method based on spherical convolution comprises the following steps:

[0083] S101: receiving a multimodal image, preprocessing the multimodal image to obtain a preprocessed multimodal image, mapping the preprocessed multimodal image to a curved surface, expanding the mapped curved surface into a spherical surface using an area-preserving deformation, and generating a spherical convolution kernel using the spherical surface;

[0084] The process of preprocessing multimodal images:

[0085] Structural MRI preprocessing steps include head motion correction, removal of non-brain tissue, pixel value normalization, spatial standardization, gray-white matter segmentation, gray-white matter interface surface mesh reconstruction, and topology correction;

[0086] Functional MRI preprocessing steps, removal of the first 10 time points, temporal layer correction, head motion correction, registration, smoothing, removal of linear drift, covariate regression, and bandpass filtering;

[0087] Diffusion MRI preprocessing steps, eddy current correction, registration, fitting of water molecule diffusion model and fiber bundle tracking;

[0088] PET preprocessing steps, partial volume effect correction; pixel value normalization;

[0089] Region of interest segmentation based on UNet and ViT;

[0090] The segmentation result is anti-aliased using a sparse field level set, and then the surface is reconstructed using the marching cubes algorithm.

[0091] The process of mapping the preprocessed multimodal image to a surface and expanding the mapped surface into a sphere using area-preserving deformation:

[0092] Mapping the tissue surface information within the neighborhood of the surface vertices to the cortex using a sphere-based mapping, a cylinder-based mapping, or a Gaussian convolution-based mapping method;

[0093] Use area-preserving deformation to expand the surface into a sphere, keeping the area of ​​the surface triangles unchanged during the expansion process;

[0094] Scale the sphere to the unit sphere and resample the vertices.

[0095] Sphere-based mapping includes:

[0096] For any vertex p on the surface, create a sphere with radius R with p as the center;

[0097] Intersect the sphere with the region of interest, and divide the intersection into one or more connected regions based on pixel connectivity;

[0098] Take the connected region containing the pixel with the closest Euclidean distance to vertex p;

[0099] The pixel information in the connected area is counted and stored in the vertex p.

[0100] The steps of cylinder-based mapping include:

[0101] For any vertex p on the surface, establish a cylinder with base radius r and height h, taking p as the base center and the surface normal vector as the axis;

[0102] Intersect the cylinder with the region of interest and divide the intersection into one or more connected regions based on pixel connectivity;

[0103] Take the connected region containing the pixel with the closest Euclidean distance to vertex p;

[0104] The pixel information in the connected area is counted and stored in the vertex p.

[0105] The mapping steps based on Gaussian convolution include:

[0106] For a vertex p on the surface, let the unit normal vector pointing to the inside of the surface be n. Given the covariance matrix of the multidimensional Gaussian distribution,

[0107]

[0108] in represents the variance along the normal vector direction, Represents the variance perpendicular to the normal vector direction;

[0109] Search for the pixel point in the area of ​​interest closest to vertex p, mark it as visited, and add the vertex to queue Q;

[0110] Assume the unit normal vector of vertex p is n, pop up the head pixel u of the Q queue, and calculate the following formula:

[0111]

[0112] Where x is a vector starting from p and ending at u, 2 is the chi-square distribution, is the preset value, L2 means taking the L2 norm of the vector within the double vertical line ||·||, that is, taking the square root of the sum of the squares of the vector elements.

[0113] If d≤1, record the pixel and its density value Mark the unvisited vertices adjacent to u as visited and add them to queue Q in sequence, ensuring that the sampled pixels are all within the 95% variance ellipsoid of vertex p;

[0114] Repeat the above steps until the queue Q is empty;

[0115] Normalize the recorded probability density;

[0116] The pixel expectations in the neighborhood of vertex p are counted based on the normalized probability density and stored at vertex p.

[0117] S102: Encode and convolve other spherical vertices within the spherical vertex area of ​​the spherical convolution kernel to obtain all spherical vertices; cluster all spherical vertices into clusters, and perform spherical pooling and de-pooling operations within the clusters;

[0118] The steps of encoding other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish the spherical convolution kernel are as follows:

[0119] like Figure 2 As shown, for the vertex p on the unit sphere, the great circle between it and the north pole of the sphere is taken as the reference line, and the great circles with angles of 45°, 90° and 135° between the tangent and the tangent of the reference line are taken;

[0120] Divide the sphere into 8 equal quadrants, then take a set of small circles with a geodesic radius of k×δ with vertex p, where k is a positive integer and k≤K, δ is a floating point number, and δ is initialized to the mean of the length of the edge of the spherical surface plus twice the standard deviation;

[0121] The small circle set and the hexagram large circle divide the neighborhood of vertex p into 8K sectors, and assign weight coefficients w0, w1, ..., w to be learned to vertex p and sector rings. 8K ;

[0122] Let the eigenvector of vertex p be h0, and let the fan ring C i There is n i vertices, among which vertex p j The eigenvector of i，j , the spherical convolution takes the mean of the feature vectors in the same fan ring and aggregates them to the vertex p:

[0123]

[0124] Where h is the feature vector of the aggregated vertex p, w0 is the coefficient to be trained for the current vertex p, and w i is the i-th sector ring C i The coefficient to be trained;

[0125] The spherical convolution kernel is the expansion of the image convolution kernel on the sphere. The fan ring is analogous to the pixel, and K and δ are analogous to the size of the image convolution kernel and the pixel size, respectively.

[0126] The process of spherical pooling and unpooling operations within a cluster:

[0127] like Figure 3 As shown, the KMeans++ method is used to randomly select M vertices on the sphere. First, the first center point is randomly selected, and the geodesic distance between it and other vertices in the area of ​​interest is calculated, which is the length of the minor arc of the great circle of the sphere.

[0128] Establish a probability distribution proportional to the square of the geodesic distance, randomly select a new center point and update the geodesic distance from the vertex to the center point until M center points are selected;

[0129] Taking the selected M vertices as the initial centers, K-center point clustering is used to assign vertices to M clusters according to the minimum geodesic distance to the center point;

[0130] Select the vertex with the smallest sum of geodesic distances to other vertices in the cluster as the new center point, and continue iterating until convergence;

[0131] Spherical pooling gathers vertex information within a cluster to the center point, and average pooling and maximum pooling are optional;

[0132] Spherical anti-pooling redistributes the center point information to other vertices in the cluster, and selects anti-average pooling and anti-maximum pooling.

[0133] S103: Receive tissue surface information, input the tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and output an auxiliary detection result, wherein the pre-established segmentation and classification spherical convolutional neural network model is established based on spherical convolution, spherical pooling and de-pooling.

[0134] The process of inputting tissue surface information into the pre-established segmentation and classification spherical convolutional neural network model and outputting auxiliary detection results:

[0135] Perform multiple spherical convolution and spherical pooling operations on the spherical surface;

[0136] Input the feature vector of the spherical vertices into the fully connected layer;

[0137] Use spherical unpooling and spherical convolution to reconstruct the spherical surface so that the input and output are the same;

[0138] Extract the bottleneck layer features of the fully connected layer and build a machine learning model as a spherical convolutional neural network model for segmentation and classification;

[0139] The encoder is constructed using multiple layers of spherical convolutional layers and spherical pooling layers to extract deep features of the tissue surface.

[0140] The spherical convolution layer can use cross-layer connections;

[0141] When the output is classification or grading, the encoder output is passed to the fully connected layer, and the fully connected layer outputs the classification or decomposition result;

[0142] When the output is semantic segmentation, a decoder is constructed using multiple layers of spherical convolutional layers and spherical unpooling layers. The decoder reconstructs the same sphere as the input, with the vertices being the labels of the semantic segmentation.

[0143] Embodiment 2: In the second aspect, as Figure 4 As shown, the embodiment of the present invention further discloses a computer-aided detection system based on spherical convolution, comprising:

[0144] The spherical surface generation module 11 is used to receive the multimodal image, preprocess the multimodal image to obtain the preprocessed multimodal image, map the preprocessed multimodal image to the curved surface, and expand the mapped curved surface into a spherical surface using area-preserving deformation;

[0145] A pooling and de-pooling module 12 is used to encode other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish a spherical convolution kernel, apply the spherical convolution to all spherical vertices; cluster all spherical vertices into clusters, and perform spherical pooling and de-pooling operations within the clusters;

[0146] The auxiliary detection module 13 is used to receive tissue surface information, input the tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and output an auxiliary detection result, wherein the pre-established segmentation and classification spherical convolutional neural network model is established based on spherical convolution, spherical pooling and de-pooling.

[0147] Specifically, the present invention is further described below by way of embodiments:

[0148] This embodiment uses the same body position to collect the three-dimensional image of the patient, and converts the original collected DICOM format image into an easy-to-process NIfTI format or npz format;

[0149] The image of one imaging modality is used as the target image, and the images of other modalities are registered to the target image using rigid deformation;

[0150] Segment the tumor region of interest, and choose the UNet segmentation model, ViT segmentation model, etc.

[0151] The segmentation result is anti-aliased using a sparse field level set, and then the surface is reconstructed using the marching cubes algorithm.

[0152] On the other hand, the present embodiment uses the same body position to acquire multimodal brain images such as three-dimensional PET, structural MRI, functional MRI and diffusion MRI of the patient, converts the original acquired DICOM format images into easy-to-process NIfTI format or npz format, and performs preprocessing respectively;

[0153] Structural MRI preprocessing: head motion correction, removal of non-brain tissue, pixel value normalization, spatial standardization, gray-white matter segmentation, gray-white matter interface mesh surface reconstruction, topology correction, etc.;

[0154] Functional MRI preprocessing: remove the first 10 time points, time layer correction, head motion correction, registration to structural MRI, smoothing, removal of linear drift, covariate regression, bandpass filtering, etc.;

[0155] Diffusion MRI preprocessing module: head motion correction, eddy current correction, registration to structural MRI, fitting water molecule diffusion model (tensor model or orientation distribution function optional), fiber bundle tracking (deterministic fiber bundle tracking or probabilistic fiber bundle tracking optional);

[0156] PET preprocessing: head motion correction, partial volume effect correction, registration to structural MRI, grayscale normalization, etc.

[0157] like Figure 5 As shown, the pre-established classification spherical convolutional neural network model can use ResNet as the backbone network: based on cortical information mapping, the information of the cortex and superficial white matter is mapped to the cortical mesh surface, and the area-preserving deformation is used to transform it into a sphere, and the ResNet constructed by the spherical convolution and spherical pooling is input; ResNet is composed of multiple spherical convolution layers and spherical pooling layers, and cross-layer residual connections are used; finally, global average pooling is used and input into the fully connected layer, and the output of the fully connected layer is the classification result;

[0158] The pre-built segmentation spherical convolutional neural network model can use UNet and UNet++ as the backbone network:

[0159] i. Encoder: Use multiple spherical convolution layers and spherical pooling layers to process and downsample the spherical surface, and the number of channels gradually increases during the downsampling process;

[0160] ii. Decoder: Use multiple spherical unpooling layers and spherical convolution layers to upsample the processed surface features, use ReLU (optionally Leaky ReLU) as the activation function, and reduce the number of channels during the upsampling process;

[0161] iii. The UNet backbone network uses cross-layer connections to concatenate the feature map channels of the same scale of the encoder and decoder, thereby combining low-level detail information with high-level semantic information;

[0162] iv. The UNet++ backbone network introduces multiple fusion modules at each level of the decoder, and realizes multi-level feature fusion by connecting feature maps from the corresponding level of the encoder and shallower levels;

[0163] The v.UNet++ backbone network embeds a series of dense cross-layer connections in the encoder and decoder sub-networks and uses a deep supervised learning method.

[0164] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0165] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0166] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0167] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A computer-aided detection method based on spherical convolution, characterized in that: The method comprises the following steps: Receiving a multimodal image, preprocessing the multimodal image to obtain a preprocessed multimodal image, mapping the preprocessed multimodal image to a surface, and expanding the mapped surface into a spherical surface using an area-preserving deformation; Encode other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish a spherical convolution kernel, and apply spherical convolution to all spherical vertices; cluster all spherical vertices into clusters, and perform spherical pooling and de-pooling operations within the clusters; The steps of encoding other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish the spherical convolution kernel are as follows: For the vertex p on the unit sphere, take the great circle between it and the North Pole of the sphere as the reference line, and take the great circles whose tangents are at angles of 45°, 90° and 135° with the tangents of the reference line; Divide the sphere into 8 equal quadrants, then take a set of small circles with a geodesic radius of k×δ with vertex p, where k is a positive integer and k≤K, δ is a floating point number, and δ is initialized to the mean of the length of the edge of the spherical surface plus twice the standard deviation; The small circle set and the hexagram large circle divide the neighborhood of vertex p into 8K sectors, and assign weight coefficients w0, w1, ..., w to be learned to vertex p and sector rings. 8K ; Let the eigenvector of vertex p be h0, and let the fan ring C i There is n i vertices, among which vertex p j The eigenvector of i，j , the spherical convolution takes the mean of the feature vectors in the same fan ring and aggregates them to the vertex p: Where h is the feature vector of the aggregated vertex p, w0 is the coefficient to be trained for the current vertex p, and w i is the i-th sector ring C i The coefficient to be trained; The spherical convolution kernel is the expansion of the image convolution kernel on the sphere, the fan ring is analogous to the pixel, K and δ are analogous to the size of the image convolution kernel and the pixel size respectively; Receive tissue surface information, input the tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and output an auxiliary detection result, wherein the pre-established segmentation and classification spherical convolutional neural network model is established based on spherical convolution, spherical pooling and de-pooling.

2. A computer-aided detection method based on spherical convolution according to claim 1, characterized in that: The process of preprocessing multimodal images: Structural MRI preprocessing steps include head motion correction, removal of non-brain tissue, pixel value normalization, spatial standardization, gray-white matter segmentation, gray-white matter interface surface mesh reconstruction, and topology correction; Functional MRI preprocessing steps, removal of the first 10 time points, temporal layer correction, head motion correction, registration, smoothing, removal of linear drift, covariate regression, and bandpass filtering; Diffusion MRI preprocessing steps, eddy current correction, registration, fitting of water molecule diffusion model and fiber bundle tracking; PET preprocessing steps, partial volume effect correction; pixel value normalization; Region of interest segmentation based on UNet and ViT; The segmentation result is anti-aliased using a sparse field level set, and then the surface is reconstructed using the marching cubes algorithm.

3. The computer-aided detection method based on spherical convolution according to claim 1, characterized in that: The process of mapping the preprocessed multimodal image to a surface and expanding the mapped surface into a spherical surface using area-preserving deformation is as follows: Mapping the tissue surface information within the neighborhood of the surface vertices to the cortex using a sphere-based mapping, a cylinder-based mapping, or a Gaussian convolution-based mapping method; Use area-preserving deformation to expand the surface into a sphere, keeping the area of ​​the surface triangles unchanged during the expansion process; Scale the sphere to the unit sphere and resample the vertices.

4. A computer-aided detection method based on spherical convolution according to claim 3, characterized in that: The sphere-based mapping includes: For any vertex p on the surface, create a sphere with radius R with p as the center; Intersect the sphere with the region of interest, and divide the intersection into one or more connected regions based on pixel connectivity; Take the connected region containing the pixel with the closest Euclidean distance to vertex p; The pixel information in the connected area is counted and stored in the vertex p.

5. The computer-aided detection method based on spherical convolution according to claim 3, characterized in that: The cylinder-based mapping step comprises: For any vertex p on the surface, establish a cylinder with base radius r and height h, taking p as the base center and the surface normal vector as the axis; Intersect the cylinder with the region of interest and divide the intersection into one or more connected regions based on pixel connectivity; Take the connected region containing the pixel with the closest Euclidean distance to vertex p; The pixel information in the connected area is counted and stored in the vertex p.

6. The computer-aided detection method based on spherical convolution according to claim 3, characterized in that: The mapping step based on Gaussian convolution includes: For a vertex p on the surface, let the unit normal vector pointing to the inside of the surface be n. Given the covariance matrix of the multidimensional Gaussian distribution, in represents the variance along the normal vector direction, Represents the variance perpendicular to the normal vector direction; Search for the pixel point in the area of ​​interest closest to vertex p, mark it as visited, and add the vertex to queue Q; Assume the unit normal vector of vertex p is n, pop up the head pixel u of the Q queue, and calculate the following formula: Where x is a vector starting from p and ending at u, 2 is the chi-square distribution, is a preset value, L2 means taking the L2 norm of the vector within the double vertical line ||·||; If d≤1, record the pixel and its density value Mark the unvisited vertices adjacent to u as visited and add them to queue Q in sequence, ensuring that the sampled pixels are all within the 95% variance ellipsoid of vertex p; Repeat the above steps until the queue Q is empty; Normalize the recorded probability density; The pixel expectations in the neighborhood of vertex p are counted based on the normalized probability density and stored at vertex p.

7. The computer-aided detection method based on spherical convolution according to claim 1, characterized in that: The process of performing spherical pooling and unpooling operations within a cluster is as follows: Use the KMeans++ method to randomly select M vertices on the sphere. First, randomly select the first center point and calculate its geodesic distance to other vertices in the area of ​​interest, which is the length of the minor arc of the great circle of the sphere. Establish a probability distribution proportional to the square of the geodesic distance, randomly select a new center point and update the geodesic distance from the vertex to the center point until M center points are selected; Taking the selected M vertices as the initial centers, K-center point clustering is used to assign vertices to M clusters according to the minimum geodesic distance to the center point; Select the vertex with the smallest sum of geodesic distances to other vertices in the cluster as the new center point, and continue iterating until convergence; Spherical pooling gathers vertex information within the cluster to the center point, and selects average pooling and maximum pooling; Spherical anti-pooling redistributes the center point information to other vertices in the cluster, and selects anti-average pooling and anti-maximum pooling.

8. The computer-aided detection method based on spherical convolution according to claim 1, characterized in that: The process of inputting tissue surface information into the pre-established segmentation and classification spherical convolutional neural network model and outputting auxiliary detection results: Perform multiple spherical convolution and spherical pooling operations on the spherical surface; Input the feature vector of the spherical vertices into the fully connected layer; Use spherical unpooling and spherical convolution to reconstruct the spherical surface so that the input and output are the same; Extract the bottleneck layer features of the fully connected layer and build a machine learning model as a spherical convolutional neural network model for segmentation and classification; The encoder is constructed using multiple layers of spherical convolutional layers and spherical pooling layers to extract deep features of the tissue surface. The spherical convolution layer uses cross-layer connection; When the output is classification or grading, the encoder output is passed to the fully connected layer, and the fully connected layer outputs the classification or decomposition result; When the output is semantic segmentation, a decoder is constructed using multiple layers of spherical convolutional layers and spherical unpooling layers. The decoder reconstructs the same sphere as the input, with the vertices being the labels of the semantic segmentation.

9. A computer-aided detection system based on spherical convolution, characterized in that: The computer-aided detection system based on spherical convolution is used to execute the computer-aided detection method based on spherical convolution as claimed in any one of claims 1 to 8, comprising: A spherical surface generation module is used to receive a multimodal image, preprocess the multimodal image to obtain a preprocessed multimodal image, map the preprocessed multimodal image to a curved surface, and expand the mapped curved surface into a spherical surface using an area-preserving deformation; The pooling and unpooling modules are used to encode other spherical vertices within the spherical vertex field of the spherical convolution kernel of the sphere to establish the spherical convolution kernel, apply the spherical convolution to all spherical vertices; cluster all spherical vertices into clusters, and perform spherical pooling and unpooling operations within the clusters; The auxiliary detection module is used to receive tissue surface information, input the tissue surface information into a pre-established segmentation and classification spherical convolutional neural network model, and output an auxiliary detection result, wherein the pre-established segmentation and classification spherical convolutional neural network model is established based on spherical convolution, spherical pooling and de-pooling.

Images