A multi-modal surface data processing optimization system based on spherical transformer

Abstract

The application provides a kind of multi-modal surface data processing optimization system based on spherical Transformer, including image acquisition and pretreatment module, spherical uniform symmetrical grid surface generation module, spherical convolution and pooling module, spherical Transformer module, spherical upsampling and downsampling module and application analysis module;The image acquisition and pretreatment module is used for segmentation of interest area and surface curve reconstruction, completes rigid registration, extracts features at the vertex of curve;The application analysis module integrates the spherical convolution and pooling module, the spherical Transformer module and the spherical upsampling and downsampling module, establishes the spherical auto-encoder model, the segmentation model and the classification model, and is used for target identification, risk prediction or anomaly detection task.The application can capture the complex pattern that traditional CNN is difficult to extract, improve the complementarity of multi-source features based on cross-modal attention mechanism, and realize efficient multi-source data fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and computer vision, and particularly relates to a multi-modal surface data processing optimization system based on a spherical Transformer. BACKGROUND

[0002] The fields of earth science, three-dimensional graphics, remote sensing images, and climate simulation often need to process a kind of "surface data", such as the surface of the earth, the surface of celestial bodies, the shell of industrial parts, and the 360° picture taken by a panoramic camera. Traditional methods usually "flatten" these curved surfaces to a planar mesh, but this will cause distortion at the edge or polar region, and it is difficult to balance global and local information. Graph neural networks and spherical convolution use mesh adjacency relationships to extract local features, but they can only see information within a "small range" and are difficult to capture information about larger-scale or global structural changes.

[0003] The recently developed Transformer method uses a global self-attention mechanism to capture long-range dependencies, but it requires the data structure to satisfy the Euclidean space assumption, which is not suitable for non-Euclidean structured surfaces. To transfer the Transformer method to spherical and other curved surface data, current research usually first subdivides the curved surface into surface patches (patches), and calculates attention between the surface patches. However, directly transferring the Transformer method in Euclidean space to non-Euclidean structured surfaces will result in a huge increase in computational load due to the large number of vertices on the curved surface, and will encounter the problem of uneven distribution of points on the curved surface. To solve the above problems, some newer methods use HEALPix and other algorithms to divide the curved surface into relatively uniform curved patches; at the same time, some research refers to the multi-modal fusion mechanism of the biological visual system to dynamically allocate attention areas. However, so far, no method can do the following at the same time:

[0004] 1. Suitable for a variety of curved surfaces, including the earth, human scans, point cloud scenes, and three-dimensional medical images, etc.

[0005] 2. Balancing local details and global structure;

[0006] 3. Reasonable computational load, capable of processing large-scale high-resolution data. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a multi-modal surface data processing optimization system based on a spherical Transformer, which can directly interface with any curved surface data and extract rich geometric features while ensuring speed, providing a unified processing capability for applications such as earth simulation, three-dimensional reconstruction, and camera panoramic understanding.

[0008] The system comprises an image acquisition and preprocessing module, a spherical uniform symmetric mesh surface generation module, a spherical convolution and pooling module, a spherical Transformer module, a spherical upsampling and downsampling module, and an application analysis module.

[0009] The image acquisition and preprocessing module is used for segmenting a region of interest and surface curve reconstruction, completing rigid registration in a unified coordinate system, and extracting features at curve vertices, especially cortex thickness, curvature, depth, multi-source signal value, signal gradient, and superficial white matter features for brain images.

[0010] The spherical uniform symmetric mesh surface generation module deforms the curve surface obtained by the image acquisition and preprocessing module into a unit sphere in a standard space, and then uses a first-order and second-order subdivision icosahedron method to resample the unit sphere into a uniform and highly symmetric graph structure, and resamples the curve surface features through natural neighborhood interpolation and radial basis function interpolation.

[0011] The spherical convolution and pooling module defines spherical convolution using spherical geometric relationships based on the uniform symmetric mesh surface obtained by the spherical uniform symmetric mesh surface generation module, and establishes a spherical pooling method based on the recursive process of the subdivision icosahedron.

[0012] The spherical Transformer module divides the spherical graph into uniform spherical graph blocks based on the spherical uniform symmetric mesh surface generation module, constructs a spherical Transformer model, and constructs a spherical cross-modal Transformer model based on a cross-modal attention mechanism.

[0013] The spherical upsampling and downsampling module realizes splitting and merging of spherical graph blocks based on the recursive process of the icosahedron subdivision method in the spherical uniform symmetric mesh surface generation module, thereby realizing upsampling and downsampling.

[0014] The application analysis module integrates the spherical convolution and pooling module, the spherical Transformer module, and the spherical upsampling and downsampling module, establishes a spherical autoencoder model, a segmentation model, and a classification model, and is used for target recognition, risk prediction, or anomaly detection tasks.

[0015] The image acquisition and preprocessing module specifically performs the following steps: segmenting the original data to extract a region of interest, and extracting a surface curve based on a marching cubes algorithm , and using a level set algorithm to expand inward or outward to obtain a vertex-to-curve surface For brain data, extract a vertex-to-curve surface of the inner surface of the gray matter and the outer surface of the gray matter The surface thickness is determined by the corresponding vertex spacing, and the intermediate surface is reconstructed at 25%, 50%, and 75% of the spacing, respectively. , and The raw data mentioned above are three-dimensional medical imaging data that reflect the anatomical structure and functional information of the nervous system.

[0016] For the same research object, a set of neural images with curved surface morphological features is acquired using multiple imaging techniques within a set time window. This set of neural images includes multi-source image data. Six-degree-of-freedom rigid registration is performed on the multi-source image data, aligning it to a reference space by minimizing mutual information or mean square error. Features are extracted at each curved surface node, including:

[0017] exist Geometric features of thickness, depth, and curvature are extracted from the surface, and the signal intensities of T1-weighted imaging (T1WI) and fluid attenuated inversion recovery (FLAIR) images based on T1 relaxation time are mapped to the vertices of the surface.

[0018] exist The surface was used as a reference to extract normalized weighted imaging T1WI and liquid attenuation inversion recovery image FLAIR, respectively.

[0019] For diffusion tensor imaging (DTI) of the brain, With each vertex on the surface as the center point, and... Corresponding points on the curved surface The truncated axisymmetric Gaussian distributions are established with the direction of the line connecting the vertices on the surface as the axis of symmetry. The intersection of the truncated axisymmetric Gaussian distribution region and the white matter is taken, and the weighted average fractional anisotropy FA, ​​average diffusivity MD, axial diffusivity AD and radial diffusivity RD are calculated.

[0020] For positron emission tomography (PET) images, The normalized ingestion values ​​corresponding to each vertex of the surface sampling are compared with the SUVR and the asymmetric exponent AI.

[0021] The spherical uniform symmetrical mesh generation module specifically performs the following steps: Based on The surface is topologically shaped into a unit sphere in standard space, while maintaining the relative spacing between the vertices of the surface and the geometry of the triangular facets within the error range and not exceeding a predetermined threshold during the deformation process;

[0022] By using registration between curved surfaces to deform the sphere to standard space and scaling it down to a unit sphere, we obtain the standard sphere. (The 2 here does not mean the square of the area or radius of the sphere, but the dimension of the sphere in n-dimensional space, which is a conventional symbol for the sphere.) Then, the subdivision icosahedron method is used to resample the unit sphere into a uniform and highly symmetrical graphic structure.

[0023] The described method for subdividing the icosahedron is an algorithm that approximates a sphere to a regular icosahedron through recursive geometric subdivision. It can obtain uniformly distributed and symmetrical vertices on the sphere. The icosahedron is a highly symmetrical Platonic polyhedron composed of 20 equilateral triangular faces, 12 vertices, and 30 edges. All 12 vertices are located on the circumscribing surface of the icosahedron and consist of the following three sets of symmetrical points:

[0024] ,

[0025] in It is the golden ratio. After normalization, all vertices lie on the standard sphere. ;

[0026] Reconstructing a uniform symmetric network on a standard sphere using the first-order and second-order icosahedral subdivision methods:

[0027] The first-order icosahedral subdivision method includes: taking the midpoints of all edges, connecting the newly added midpoints to all faces to divide the original equilateral triangle into 4 equilateral triangles, and then normalizing the coordinates of the newly added midpoints so that they fall on a standard sphere; after a single first-order subdivision, the number of faces S increases to 4*S, and the number of edges E increases to The number of points V increases to V+E.

[0028] The second-order regular icosahedral subdivision method includes: assuming the edge length is... Take the distance from the existing vertex. The point is selected, and the center point of the equilateral triangle is chosen. Each equilateral triangle face is divided into 9 smaller equilateral triangles. The coordinates of the newly added points are normalized so that the new points fall on the standard sphere. After a single second subdivision, the number of faces S increases to 9*S, and the number of edges E increases to The number of points V increases to ;

[0029] Apply the first-order regular icosahedral subdivision method and the second-order regular icosahedral subdivision method more than twice, or alternately apply the first-order regular icosahedral subdivision method and the second-order regular icosahedral subdivision method more than twice, and set the last application to be the second-order regular icosahedral subdivision method.

[0030] After obtaining a spherical uniform symmetric grid, the features extracted by the image acquisition and preprocessing module are interpolated using natural neighborhood interpolation or radial basis function interpolation.

[0031] The natural neighborhood interpolation includes: the original set of sampling points on the sphere. Calculate the spherical Voronoi diagram, for the i-th original sampling point. Voronoi unit Including distances on the sphere The nearest vertex, where i takes values ​​from 1 to n. , where x represents any vertex on the sphere;

[0032] This represents the geodesic distance; the spherical vertex q obtained by the icosahedral subdivision method is added to the point set P, and the Voronoi diagram is recalculated, and the Voronoi elements of the spherical vertex q are identified. All satisfy The original vertex of vertex q is called the natural neighborhood of vertex q. The contribution weights of the natural neighborhood vertices to vertex q are calculated. Where Area represents the area; the eigenvalues ​​of vertex q Original sampling points within the natural neighborhood of vertex q eigenvalues The weighted summation is used to obtain the result. The feature value Characterizing the original sampling points The local physical or physiological properties of the location, which have been determined before interpolation;

[0033] The radial basis function interpolation method includes: for any vertex q of a sphere obtained by the icosahedral subdivision method, taking the original sampling points in the neighborhood of vertex q. , making ,in, The standard deviation of the Gaussian kernel function; weights are calculated based on the Gaussian kernel function. Where exp represents the natural exponential function; and the eigenvalues ​​of vertex q are... Original sampling points within the neighborhood eigenvalues The weighted average;

[0034] in, .

[0035] The spherical convolution and pooling module specifically performs the following steps:

[0036] Let u be the spherical distance between adjacent vertices. Spherical convolution will transform the current vertex... of The vertex information in the neighborhood is integrated, wherein is the radius of the spherical convolution kernel; and the current vertex is the distance between the current vertex and the vertex , and Each of the vertices of the icosahedron has 6 vertices, and is uniformly distributed on 3 concentric circles; the spherical convolution kernel is based on the meridian line passing through the current vertex , and the convolution weight is assigned in the clockwise direction, and the distance between the vertex and the 12 vertices of the icosahedron is There are 5 vertices, and the average of the 5 vertices is taken as the filling;

[0037] A spherical pooling method is established based on the recursive process of the subdivided icosahedron: the first-order subdivided icosahedron corresponds to a pooling method, which includes: setting the current subdivision recursive number as l, the center vertex and the outer hexagon vertex are the spherical vertex after l-1 recursive, and the remaining vertex is the newly generated vertex of this recursive; the spherical pooling integrates the information of the center vertex and all the vertices adjacent to the center vertex in the first order into the center vertex, and removes the newly generated vertex in this recursive;

[0038] The second-order subdivision method corresponds to a pooling method, which includes: integrating the information of the newly generated vertex adjacent to the center vertex and the center vertex of the surrounding triangle into the center vertex.

[0039] The spherical Transformer module specifically performs the following steps:

[0040] Spherical graph block division: taking the vertex of the second-to-last icosahedron subdivision method as the center point, and taking the adjacent second-order subdivision method vertices together to form a graph block; in addition to the 12 vertices of the icosahedron, each center point has 12 adjacent vertices, of which 6 are located at the adjacent vertices of the triangle center and are shared with the surrounding graph blocks to capture the context information between adjacent blocks; the icosahedron vertex has only 10 adjacent vertices, including 5 edge center points and 5 triangle face center points; the mean values of the edge and triangle face center points are taken respectively to fill the vectors; after the spherical graph block is flattened, linear projection is performed, and each graph block is mapped to a feature matrix :

[0041] ,

[0042] Wherein represents the real field, d is the vertex feature dimension, and each spherical graph block has 13 vertices.

[0043] The spherical Fourier position is used to encode the position of the spherical graph block:

[0044] ,

[0045] Wherein respectively the zenith and azimuth of the spherical polar coordinate of the vertex, is a learnable parameter matrix; PE represents the position encoding; k represents the index variable of summation; represents the tensor product, different vectors, matrices or other mathematical objects can be combined in a certain rule to generate more complex mathematical objects to express more abundant information or perform specific operation;

[0046] Establish a spherical Transformer model: for the feature vector h of the spherical graph block, apply the sub-attention mechanism:

[0047] ,

[0048] wherein Attention represents the attention mechanism, Q, K and V are respectively the query vector, the key vector and the value vector, which are obtained by matrix multiplication of the feature vector h with the query matrix, the key matrix and the value matrix; T represents the matrix transfer; the spherical Transformer model performs multiplication on the attention and the feature vector h, then performs layer normalization, inputs the feedforward layer for comprehensive processing, and then performs layer normalization again;

[0049] Establish a spherical cross-modal Transformer model: the spherical cross-modal Transformer model comprises a multi-head attention module with D layers of cross-modal attention modules , for the feature vector of multi-source data and the feature vector of multi-source data , wherein and The difference between them is: provides the source data feature, which is used to generate the key and value vectors to capture the inherent mode of the data; provides the target data feature, which is used to generate the query vector to guide the attention allocation, the cross-modal attention module transmits the information in into :

[0050] ,

[0051] wherein the query vector of is obtained from the feature vector , the key vector and the value vector of are obtained from , , and to-be-trained parameters;

[0052] For the multi-head attention module with D-layer cross-modal attention modules , the cross-layer connection is used in each layer to perform a first cross-layer summation with the output of the cross-modal attention module , then perform layer normalization, and then input into a feedforward layer for nonlinear deformation, and then perform a second cross-layer summation and layer normalization, wherein the cross-layer summation refers to an element-wise addition operation of the input feature vector of the current layer and the output feature vector processed by the cross-modal attention module, which is used to retain the original input information in the feature fusion process, while introducing new features processed to enhance the expression ability and training stability of the model.

[0053] The spherical upsampling and downsampling module comprises a downsampling layer and an upsampling layer;

[0054] The downsampling layer uses a recursive process based on the subdivision icosahedron method to merge spherical graphic blocks, and the center point of the initial state graphic block is the vertex obtained by the last first-order subdivision method, the mean value of the feature vectors of all vertices in the graphic block is taken and converged at the midpoint of the graphic block, and the weighted average of all vertices in the block is taken, and the weight is determined by the proportion of the area of the spherical Voronoi cell:

[0055] ,

[0056] wherein, is the area of the Voronoi region of the vertex i, which is calculated by recursive subdivision level; represents the sum of the areas of all Voronoi regions in the graphic block; represents the feature vector of the i-th vertex; represents the output feature vector obtained by weighted pooling, which is the aggregated representation of the image block; N represents the total number of vertices;

[0057] The feature mean value of the vertex i and the newly sampled vertices of the current level adjacent to the vertex i is taken as the convergence target of the above level subdivision method, and the downsampling process reversely simulates the subdivision recursive step.

[0058] The upsampling layer uses a recursive process based on the subdivision icosahedron method to reconstruct the vertex, starting from the vertex of the last icosahedron subdivision method, and backtracking to the initial icosahedron structure layer by layer, and the parent vertex of each level is determined by the subdivision tree topology; the low-resolution features are restored to high-resolution grids by linear interpolation:

[0059] Performing first-order feature interpolation: for each edge of the equilateral triangle, the two endpoints correspond to two original vertices A and B, and the feature vectors of the two original vertices A and B are and , and and take the mean value to generate the edge midpoint feature :

[0060] ,

[0061] The first-order feature interpolation corresponds to the projection of the set of icosahedral first-order subdivision; the set of icosahedral subdivision refers to the set of two or more hierarchical grid structures generated by recursively subdividing the initial icosahedron, each level corresponding to the vertex and face topology formed by one subdivision operation;

[0062] Performing second-order feature extension: inserting two new vertices at the trisection point of each edge, and calculating the features of the two new vertices by linear interpolation and :

[0063] ,

[0064] ,

[0065] Subsequently, the calculated and , i.e., the mean value of the features of the six edge vertices, are used as the face center point features.

[0066] The application analysis module is used in the following application scenarios:

[0067] Spherical autoencoder extracts depth features: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, and the spherical uniform symmetric grid surface generation module is used to establish a spherical grid, then the spherical Transformer module and down-sampling layer are used to establish an encoder, and the spherical convolution, spherical pooling and up-sampling layer are used to establish a lightweight decoder, the mean square error is used as the loss function, and the output of the encoder is the depth feature of the data;

[0068] Fusion of multi-element data segmentation model: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, the spherical uniform symmetric grid surface generation module is used to establish a spherical grid, the spherical convolution and pooling module is used to establish a spherical encoder, and the depth spherical features of multi-source data are extracted respectively, the spherical Transformer module is used to fuse the features, and the spherical convolution and pooling module, the spherical up-sampling and down-sampling module are used to establish a U-Net network to realize data segmentation;

[0069] The classification model of fused multi-source data: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, a spherical uniform symmetrical grid surface generation module is used to establish a spherical grid, a spherical convolution and pooling module is used to establish a spherical encoder, and the deep spherical features of multi-source data are extracted respectively, a spherical Transformer module is used to fuse the features, and after splicing, a ResNet network is established by using the spherical convolution and pooling module to realize data classification.

[0070] The application further provides an electronic device comprising a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the system.

[0071] The application further provides a storage medium storing computer programs or instructions, and when the computer programs or instructions are run on a computer, the system is executed.

[0072] The application has the following beneficial effects: 1. A mapping process from raw data to a spherical uniform symmetrical grid is constructed, a regular icosahedron grid is used to ensure uniform distribution of each vertex, the topological structure of the curved surface can be effectively preserved, and the subsequent data processing calculation complexity can be reduced by using spherical geometric consistency.

[0073] 2. The attention mechanism of the Transformer can model long-range dependencies, capture complex patterns that are difficult to extract by traditional CNNs, and improve the complementarity of multi-source features through cross-modal attention mechanisms to achieve efficient multi-source data fusion. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 It is a structural block diagram of a multi-modal surface data processing optimization system based on a spherical Transformer according to an embodiment of the application.

[0075] Figure 2 It is a spherical regular icosahedron subdivision method according to an embodiment of the application, including first-order and second-order subdivision method schematic diagrams.

[0076] Figure 3 It is a spherical convolution and spherical pooling method schematic diagram according to an embodiment of the application.

[0077] Figure 4 It is a spherical cross-modal Transformer structure diagram according to an embodiment of the application.

[0078] Figure 5 It is a network diagram of a spherical autoencoder extracting curved surface depth features according to an embodiment of the application.

[0079] Figure 6 It is a spherical segmentation model schematic diagram of fused multi-source data according to an embodiment of the application. DETAILED DESCRIPTION

[0080] The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings in which:

[0081] As Figure 1 shown, the embodiment of the present application provides a multi-modal surface data processing optimization system based on spherical Transformer, comprising the following modules:

[0082] An image acquisition and preprocessing module is used to segment the region of interest and reconstruct the surface curve, complete rigid registration in a unified coordinate system, and extract features at the vertex of the curve, especially for brain images to extract cortical thickness, curvature, depth, multi-source signal value, signal gradient, and superficial white matter features.

[0083] The specific implementation is as follows: segment the original data to extract the region of interest, and extract the surface curve based on the marching cubes algorithm , and use the level set algorithm to expand inward or outward to obtain the vertex-to-surface curve For brain data, extract the vertex-to-surface curve of the inner surface of the gray matter and the outer surface curve , take the distance between the corresponding vertices as the thickness of the curve, and reconstruct the intermediate curve at the 25%, 50%, and 75% intervals, respectively , and ; the original data is three-dimensional medical image data reflecting the anatomical structure and functional information of the nervous system.

[0084] For the same research object (referring to a specific subject or experimental sample), a set of neural images with curved surface morphology characteristics is obtained within a set time window (such as the same or similar time window) (the specific time range is determined according to the research needs, for example, referring to the data obtained within 48 hours, a uniform standard is specified within a research) by multiple imaging techniques (such as magnetic resonance imaging (MRI), positron emission tomography (PET), etc. The present application does not limit the specific imaging techniques used, researchers can select the corresponding imaging method according to the characteristics of the required data, and these imaging techniques belong to the technical means of the data preparation stage, and are not the core content of the present application. ) The neural image set includes multi-source image data, which describes the characteristics of neural tissue from different imaging modalities (such as structure, function, or metabolism, etc.). Six degrees of freedom rigid registration is performed on the multi-source image data to align to the reference space with mutual information or minimum mean square error, and features are extracted at each curve node, including:

[0085] In The geometric features of the curved surface, including thickness, depth, and curvature, are extracted, and the signal intensity of T1-weighted imaging (T1WI) and fluid-attenuated inversion recovery image (FLAIR) based on T1 relaxation time (i.e., the speed of magnetic resonance signal recovery) is mapped to the vertex of the curved surface;

[0086] In The curved surface, with the average signal value of the curved surface as the reference, extracts the normalized T1-weighted imaging (T1WI) and fluid-attenuated inversion recovery image (FLAIR), respectively;

[0087] For the brain diffusion tensor imaging image (DTI), the curved surface is extracted from the image, and the geometric features of the curved surface, including thickness, depth, and curvature, are extracted. Each vertex on the curved surface is taken as the center point, and the direction of the line connecting the corresponding point on the curved surface to the vertex is taken as the symmetry axis to establish a truncated axisymmetric Gaussian distribution. The direction of the line connecting the corresponding point on the curved surface to the vertex is taken as the symmetry axis to establish a truncated axisymmetric Gaussian distribution. The intersection of the truncated axisymmetric Gaussian distribution region and the white matter is taken, and the weighted average fractional anisotropy (FA), mean diffusivity (MD), axial diffusivity (AD), and radial diffusivity (RD) are calculated.

[0088] For the positron emission tomography image (PET), the curved surface is extracted from the image, and the geometric features of the curved surface, including thickness, depth, and curvature, are extracted. The curved surface samples the standardized uptake value ratio (SUVR) and the asymmetry index (AI) corresponding to each vertex.

[0089] The spherical uniform symmetric mesh surface generation module: based on the image acquisition and preprocessing module, the spherical surface is generated from the image data. The curved surface is deformed to a unit sphere in the standard space, and the relative distance between the vertices and the geometric shape of the triangular patches are maintained within the error range and do not exceed the predetermined threshold (e.g., the distance between the vertices changes by no more than 5%).

[0090] Then the spherical surface is deformed to the standard space using the inter-surface registration, and is scaled to a unit sphere, which is the standard spherical surface . The unit sphere is resampled into a uniform and highly symmetric graph structure using the subdivision icosahedron method.

[0091] The subdivision icosahedron method is an algorithm that approximates a sphere by recursively geometrically partitioning an icosahedron, which can obtain uniformly distributed and symmetric vertices on the spherical surface. The icosahedron is a highly symmetric Platonic solid composed of 20 equilateral triangular faces, 12 vertices, and 30 edges, with all 12 vertices located on its circumscribed sphere. It is composed of the following three groups of symmetric points:

[0092] ,

[0093] where ; after normalization, all vertices are located on the standard spherical surface . For example, Figure 2The uniform symmetric network is reconstructed on the standard sphere using the first-order and second-order icosahedral subdivision method as shown in the following:

[0094] First-order icosahedral subdivision method: For all edge midpoints, the original equilateral triangle is divided into four equilateral triangles by connecting the new midpoints. Then the coordinates of the new midpoints are normalized to fall on the standard sphere. After a single first-order subdivision, the number of faces S increases to 4*S, the number of edges E increases to , and the number of points V increases to V+E.

[0095] Second-order icosahedral subdivision method: Let the edge length be , take the point with a distance of from the existing vertex, and take the center point of the equilateral triangle to divide each equilateral triangle into nine small equilateral triangles. Normalize the coordinates of the new points to fall on the standard sphere. After a single second-order subdivision, the number of faces S increases to 9*S, the number of edges E increases to , and the number of points V increases to ; for fast calculation, when , take .

[0096] Multiple applications of the first-order subdivision method, the second-order subdivision method, or alternating applications, with the forced requirement of the last application of the second-order icosahedral subdivision method. After obtaining the uniform symmetric grid on the sphere, the natural neighborhood interpolation or the radial basis function interpolation method is applied to interpolate the features extracted by the image acquisition and preprocessing module.

[0097] Natural neighborhood interpolation: For the original sampling point set on the sphere, the Voronoi diagram is calculated, and the Voronoi cell of each original vertex contains the nearest vertex on the sphere to , , , which represents the geodesic distance. The sphere vertex q obtained by the icosahedral subdivision method is added to the point set and the Voronoi diagram is recalculated, and the Voronoi cell of point q is identified. All original vertices that satisfy are called the natural neighborhood of point q, and the Sibson weight is calculated, where Area represents the area. The feature value of point q is obtained by weighted summation calculation of the feature values of the vertices in its natural neighborhood , . .

[0098] Radial basis function interpolation: For any sphere vertex q obtained by the icosahedral subdivision method, take the original sampling points in its neighborhood, so that ,in Indicates geodesic distance. Let be the standard deviation of the Gaussian kernel function. Calculate the weights based on the Gaussian kernel function. The eigenvalues ​​of point q are the vertices in its neighborhood. eigenvalues The weighted average, .

[0099] Spherical convolution and pooling module:

[0100] Let u be the spherical distance between adjacent vertices. Spherical convolution will transform the current vertex... of The vertex information within the neighborhood is synthesized, among which Let be the radius of the spherical convolution kernel. For example... Figure 3 As shown in the left figure, the distance from the current vertex p , Both 2u and 2u have 6 vertices, evenly distributed on 3 concentric circles. The purple vertex represents the current vertex, and the red, blue, and green colors represent radii. , and The convolution kernel adds a new vertex. The spherical convolution kernel passes through the current vertex. Using the meridian as a reference, convolution weights are distributed clockwise. Furthermore, due to the Gaussian curvature of the spherical surface, the distance between the convolution weights and the 12 vertices of the icosahedron is... There are only 5 vertices, and their average value is used as the filler.

[0101] The spherical pooling method is based on the recursive process of subdividing a regular icosahedron. The pooling method corresponding to a first-order subdivided regular icosahedron is as follows: Figure 3 As shown in the middle diagram, if the current recursion count of the subdivision method is l, then the central red vertex and the outer hexagonal blue vertex are the spherical vertices after l-1 recursions, and the remaining vertices are the newly generated vertices in this recursion. Spherical pooling gathers the information of the vertices within the red circle (the inner hexagonal green vertex and the red center point) to the center point and removes the newly generated vertices in this recursion. The pooling method corresponding to the second-order subdivision method is as follows: Figure 3 As shown in the right figure, the information of newly generated vertices adjacent to the center point and the center points of surrounding equilateral triangles (green vertices) are converged to the center point.

[0102] Spherical Transformer module:

[0103] Icosahedron subdivision: The vertices of the second-to-last icosahedron subdivision are taken as the center points, and the adjacent vertices of the second subdivision are taken as the graphic blocks. In addition to the 12 vertices of the icosahedron, each center point has 12 adjacent vertices, of which 6 are adjacent vertices located at the center of the equilateral triangle and shared with the surrounding graphic blocks to capture the context information between adjacent blocks. Due to the existence of Gaussian curvature on the sphere, it is not possible to form a hexagonal tiling similar to two-dimensional, and the icosahedron vertex only has 10 adjacent vertices, including 5 edge center points and 5 equilateral triangle face center points. In order to make the vector length consistent after the sphere graphic block is flattened, the mean value of the edge and equilateral triangle face center points is taken to fill the vector. After flattening the sphere graphic block, linear projection is performed, and each graphic block is mapped to a feature matrix :

[0104]

[0105] where represents the real field, d is the feature dimension of the vertex, and each sphere graphic block has 13 vertices.

[0106] Position encoding: The sphere Fourier position is used to encode the position of the sphere graphic block:

[0107]

[0108] where are the spherical polar coordinates of the vertex zenith angle and azimuth angle, is a learnable parameter matrix.

[0109] Sphere Transformer model: For the feature vector h of the sphere graphic block, the sub-attention mechanism is applied:

[0110]

[0111] where Q, K and V are query vector, key vector and value vector, respectively, obtained by matrix multiplication of feature vector h with query matrix, key matrix and value matrix. The sphere Transformer model performs layer normalization after multiplying the attention with the feature vector h, and then inputs the feedforward layer for integration, and then performs layer normalization again.

[0112] Sphere cross-modal Transformer model: As shown in Figure 4 , for the feature vectors and representing multi-source data and , the cross-modal attention module transmits the information in to : ​​​

[0113] ,

[0114] The query vector From the eigenvector Obtain the key vector Sum value vector Depend on get, Let be the dimension of the key vector. , and These are the parameters to be trained. Cross-modal Transformer, such as... Figure 4 As shown in the right figure, including layer The multi-head attention module uses cross-layer connections in each layer to... and After summing the output, layer standardization is performed. Then, the input to the feedforward layer undergoes nonlinear deformation, followed by a second cross-layer summation and layer standardization.

[0115] Spherical sampling and downsampling module:

[0116] Downsampling layer: The downsampling layer merges spherical graphic blocks based on the recursive process of the subdivision icosahedron method. The center point of the initial graphic block is the vertex obtained by the last first-order subdivision method. The mean of the feature vectors of all vertices in the graphic block is taken and converged at the center point of the graphic block. A weighted average is taken for all vertices in the block, and the weights are determined by the area ratio of the spherical Voronoi unit.

[0117] ,

[0118] in Let i be the area of ​​the Voronoi region of vertex i, which is calculated recursively through subdivision levels. Taking the vertices of the previous level of subdivision as the convergence target, the feature mean of the vertices of the previous level and the adjacent newly sampled vertices of the same level is taken, and the downsampling process simulates the subdivision recursive steps in reverse.

[0119] Upsampling layer: The upsampling layer reconstructs vertices based on the recursive process of the icosahedral subdivision method. Starting from the vertex of the last icosahedral subdivision, it backtracks upwards layer by layer to the initial icosahedral structure. The parent vertex of each level is determined by the subdivision tree topology. Low-resolution features are then restored to a high-resolution mesh using linear interpolation.

[0120] First-order feature interpolation: The mean value of the two original fixed-point features of each equilateral triangle side is used to generate the side midpoint feature.

[0121] ,

[0122] This operation corresponds to the projection of the midpoint of the set of first-order subdivisions of a regular icosahedron.

[0123] Second-order feature expansion: two new vertices are inserted at the trisection point of each edge, and the features are calculated by linear interpolation:

[0124]

[0125]

[0126] Subsequently, the average of the features of the six edge vertices is taken as the feature of the face center point.

[0127] Application analysis module: there are multiple application methods for different task requirements, and the following is an example of three application scenarios:

[0128] Spherical autoencoder extracts deep features: as shown in Figure 5 , the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, and the spherical uniform symmetric grid surface generation module is used to establish a spherical grid. Then the Transformer module and down-sampling layer are used to establish an encoder, and the spherical convolution, spherical pooling and up-sampling layer are used to establish a lightweight decoder, and the mean square error is used as the loss function. The output of the encoder is the deep feature of the data. In Figure 5 , N is a natural number, indicating that the spherical grid is divided into N sub-regions (patches) and input into the Transformer model for processing. The specific value of N depends on the different tasks, i.e. the number of vertices that the spherical surface is subdivided into. In this flowchart, N is an unknown quantity representing the universality of the method, and is intended to demonstrate the processing flow of the invention, rather than a fixed value.

[0129] Segmentation model fusing multi-element data: as shown in Figure 6 , the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, and the spherical uniform symmetric grid surface generation module is used to establish a spherical grid. The spherical convolution and pooling module is used to establish a spherical encoder to extract the deep spherical features of multi-source data, and then the spherical Transformer module is used to fuse the features, and the U-Net network is established by using the spherical convolution and pooling module, the up-sampling module, etc. to realize data segmentation.

[0130] Classification model fusing multi-source data: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, and the spherical uniform symmetric grid surface generation module is used to establish a spherical grid. The spherical convolution and pooling module is used to establish a spherical encoder to extract the deep spherical features of multi-source data, and then the spherical Transformer module is used to fuse the features, and the ResNet network is established by using the spherical convolution and pooling module to realize data classification.

[0131] ​​In this embodiment, the method is applied to the classification task of brain imaging data. Each data sample includes T1-weighted magnetic resonance imaging (T1WI), liquid attenuated inversion recovery sequence (FLAIR), diffusion tensor imaging (DTI), functional magnetic resonance imaging (fMRI), and positron emission tomography (PET). The method uses a spherical uniform symmetric mesh generation module, a spherical convolution and pooling module, a spherical Transformer module, and spherical sampling and downsampling modules to construct a spherical Transformer network to achieve the brain imaging data classification task.

[0132] Step 1, Image Acquisition and Preprocessing;

[0133] The raw 3D brain image data underwent preprocessing, including denoising, eddy current correction, brain region extraction, and gray-white matter segmentation. Rigid registration was then performed on the multimodal image data, deforming it to T1WI images. The inner surface of the gray matter (i.e., the gray-white matter interface) was used as the surface. The outer surface of the gray matter is taken as a curved surface. and on the curved surface and Reconstruct the intermediate surface at 25%, 50%, and 75% of the vertex spacing. , and The thickness is defined as the Euclidean distance between corresponding vertices, with an average thickness ranging from 2 mm to 4 mm.

[0134] Feature extraction at each surface node: Geometric features were extracted from the surface, including thickness (range 0 mm to 5 mm), depth (range 0 mm to 20 mm), and curvature (range -0.5 to 0.5); T1WI and FLAIR signal intensities were mapped to the surface vertices and normalized to the range of 0 to 1; to Between all curved surfaces, with The average signal value of the surface is used as a reference to extract normalized T1WI and FLAIR values; in Normalized T1WI and FLAIR signal gradients were extracted from the curved surface; fMRI was used to extract these gradients. Local heterogeneity (ReHo) and fractional low-frequency fluctuation amplitude (fALFF) are extracted from the surface, and the mean is taken between corresponding vertices; for PET images, the average uptake rate (SUVR) is calculated with the cerebellum or pons average uptake value as a reference. The curved surface extraction SUVR (range 0 to 2) is averaged between the corresponding vertices, and the asymmetry index AI = 2 x | left SUVR - right SUVR | / (left SUVR + right SUVR) is calculated, with a range of 0 to 1, where left SUVR represents the average uptake rate on the left side of the human brain, and right SUVR represents the average uptake rate on the right side of the human brain; for the DTI image, the truncated axial symmetric Gaussian distribution (standard deviation σ between 1~3mm) is constructed with the curved surface vertex as the center and the corresponding connecting line direction as the symmetry axis, and the weighted average anisotropy fraction (FA, range 0 to 1), average diffusion rate (MD, range 0 to 0.002mm² / s), axial diffusion rate (AD, range 0 to 0.002mm² / s), and radial diffusion rate (RD, range 0 to 0.002mm² / s) are calculated. The curved surface vertex is centered, and the corresponding connecting line direction is the symmetry axis to construct a truncated axial symmetric Gaussian distribution (standard deviation σ between 1~3mm), and the weighted average anisotropy fraction (FA, range 0 to 1), average diffusion rate (MD, range 0 to 0.002mm² / s), axial diffusion rate (AD, range 0 to 0.002mm² / s), and radial diffusion rate (RD, range 0 to 0.002mm² / s) are calculated.

[0135] The input is the original multi-modal image, for example, the T1WI volume of one sample is 256x256x176 voxels. The output is a set of curved surfaces for each sample (about 150,000 vertices) to , with about 150,000 vertices, and each vertex is associated with a 23-dimensional feature vector (3-dimensional geometric features + 6-dimensional T1WI + 6-dimensional FLAIR + 2-dimensional fMRI + 2-dimensional PET + 4-dimensional DTI). The registration error is calculated by the root mean square error (RMSE) to be less than 0.5mm.

[0136] Step 2, curved surface deformation and resampling;

[0137] Any curved surface is mapped to a standard unit sphere through topological deformation and registration, with an iteration number between 150~250, and the error of maintaining the relative distance between vertices and the shape of triangular faces is less than 5%. A regular icosahedron is used as a seed, and a recursive first-order subdivision method is used to generate a uniform and highly symmetric mesh (level=4, generating 2,562 vertices).

[0138] Interpolation of pre-extracted vertex features: natural neighborhood interpolation is used, and the natural neighborhood of the new point q is found through the spherical Voronoi diagram (neighborhood number k=6), and the neighborhood features are summed according to the Sibson weight.

[0139] Input: surface and features of step 1. Output: uniform spherical mesh (symmetry error less than 3%), interpolated continuous 23-dimensional feature field. Mesh coverage is 99%, and interpolation error is less than 0.03 by L2 norm. For example, a thickness feature of an Alzheimer's disease (AD) sample varies (variance) about 0.2 (normal person about 0.1, suggesting possible abnormalities) in the frontal lobe area (the front of the brain, responsible for thinking) on the sphere. The continuous field formed by the anisotropy fraction (DTI FA) measured by diffusion tensor imaging technology (like a map of white matter pathways) shows that the white matter tracks (neural signal transmission pathways) are interrupted, which may be a sign of white matter damage.

[0140] Step 3, spherical convolution and pooling;

[0141] Integrate vertex information through spherical convolution operation: set the spherical distance between adjacent vertices as u (average vertex distance about 0.02), and in the neighborhood of the current vertex p with radius u, aggregate information with vertices with distance u from p (6, uniformly distributed on concentric circles). For example, Figure 3 The left figure shows that the vertices with distance , , and from the current vertex p are 6, and are uniformly distributed on 3 concentric circles. The purple vertex represents the current vertex, and the red, blue, and green colors represent the new vertices of the convolution kernel with radii , and . The convolution kernel is assigned weights in a clockwise direction based on the meridian passing through p, and the mean value of the 5 adjacent points in the icosahedron with an average distance of u is filled.

[0142] Implement spherical pooling based on the recursive process of subdivided icosahedron: as shown in the middle figure of Figure 3 , the first-order subdivision pooling aggregates all the adjacent information of the center vertex and its outer hexagon vertices (belonging to the l-1 level) to the center vertex and removes the newly generated vertices. The model uses 2 layers of convolution (channel number increases from 16 to 32, and activation function is ReLU) and 1 layer of pooling. In addition, the second-order subdivision method corresponds to the pooling method as shown in the right figure of Figure 3 , which aggregates the information of the newly generated vertices adjacent to the center vertex and the center vertices of the surrounding equilateral triangles (green vertices) to the center vertex.

[0143] Input is the spherical feature of convolution (2,562 vertices x 8 dimensions). Output is the compressed feature map (vertices reduced to 642, dimension 32). The post-convolution feature aggregation enhances the local contrast, for example, the curvature peak of AD samples is enhanced by about 10%~15%. The pooling compression rate reaches 70%~75%, retaining key geometric information such as thickness gradient. The standard deviation of the output feature map is about 0.1 for normal control NC samples, while AD samples show abnormal clustering.

[0144] Step 4, spherical cross-modal Transformer fusion of multi-modal features;

[0145] Spherical graph block division: taking the last second-order icosahedral subdivision vertex as the center, the adjacent second-order subdivision vertices are taken to form a graph block. For example, Figure 4 As shown in the left figure, each center point has 12 adjacent points (6 shared with adjacent blocks), and the icosahedral vertex has 10 adjacent points. The mean filling ensures the consistency of the vector length.

[0146] Cross-modal Transformer model: as shown in the right figure, the T1WI and DTI features are fused, and the cross-modal attention module transmits the DTI information into the T1WI feature, including 4 layers of multi-head attention, each layer is processed by layer normalization and feedforward layer. The input is the compressed feature of step 3, and the output is the preliminary fused 32-dimensional feature matrix. Figure 4

[0147] Step 5, spherical upsampling and downsampling;

[0148] Downsampling layer: based on icosahedral subdivision recursion, merging graph blocks, taking the mean of vertex features to converge at the midpoint. From level=4 to level=2, the number of vertices is reduced from 642 to 162. As shown in the figure, Figure 5 Downsampling is used to encode the compressed feature.

[0149] Upsampling layer: from level=2 back to level=4, linear interpolation is used to restore the feature, and the edge midpoint takes the mean of the two ends. As shown in the figure, Figure 5 Upsampling is used to reconstruct the feature in the decoder. The input is the preliminary fused feature of step 4, and the output is the resolution-adjusted feature field with 642 vertices.

[0150] Step 6, application analysis (classification task);

[0151] ​The aforementioned spherical uniform symmetric grid is taken as a unified representation framework to construct a multi-modal analysis system: a spherical convolution-pooling module extracts features, a Transformer module fuses T1WI and DTI data, and a sampling module adjusts the resolution; the classification head includes a global average pooling followed by a fully connected layer, outputs 2-class labels, and uses a softmax activation function. Training uses the PyTorch framework, an Adam optimizer (learning rate of 0.001), a cross-entropy loss function, a batch size of 32, and 50 training rounds. The data set is divided as follows: 80% for training (160 samples) and 20% for testing (40 samples). As shown in Figure 6 the spherical convolution and pooling module is used to construct a ResNet after fusing the features.

[0152] The input is multi-modal spherical features. The output is a classification label (AD or NC). The training loss decreases from an initial 1.2 to 0.3-0.4 (converges at the 30th-40th round). For example, the output probability of an AD test sample is: AD is 0.80-0.85, NC is 0.15-0.20; the output probability of an NC sample is: AD is 0.10-0.15, NC is 0.85-0.90. The overall accuracy of the test set is between 80%-85%, the precision is between 79%-84%, the recall is between 81%-86%, and the F1 score is between 0.80-0.85. The confusion matrix shows that 15-17 / 20 AD samples are correctly classified, and 16-18 / 20 NC samples are correctly classified.

[0153] The method solves the problems of topological distortion and multi-modal alignment in brain image processing, improves the classification accuracy, and maintains the rotational equivariance and geometric consistency. In real applications, the method can be used for complex data and analysis screening.

[0154] The application provides a multi-modal surface data processing optimization system based on a spherical Transformer, and there are many methods and approaches to achieve the technical scheme, and the preferred embodiments described above are only preferred embodiments of the application. It should be noted that for ordinary skilled persons in the technical field, without departing from the principle of the application, some improvements and refinements can be made, and these improvements and refinements should be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be realized by using existing technology.

Claims

1. A spherical Transformer-based multi-modal surface data processing optimization system, characterized in that, The application comprises an image acquisition and preprocessing module, a spherical uniform symmetric grid surface generation module, a spherical convolution and pooling module, a spherical Transformer module, a spherical upsampling and downsampling module, and an application analysis module. The image acquisition and preprocessing module is used for segmenting a region of interest and reconstructing a surface curve, completing rigid registration in a unified coordinate system, and extracting features at a curve vertex. The spherical uniform symmetric grid surface generation module deforms a curve obtained by the image acquisition and preprocessing module into a unit sphere in a standard space, resamples the unit sphere into a uniform and highly symmetric graph structure using a first-order and second-order subdivision icosahedron method, and resamples the curve features through natural neighborhood interpolation and radial basis function interpolation. The spherical convolution and pooling module defines spherical convolution using spherical geometric relationships based on the uniform symmetric grid surface obtained by the spherical uniform symmetric grid surface generation module, and establishes a spherical pooling method based on a recursive process of the subdivision icosahedron. The spherical Transformer module divides a spherical graph into uniform spherical graph blocks based on the spherical uniform symmetric grid surface generation module, constructs a spherical Transformer model, and constructs a spherical cross-modal Transformer model based on a cross-modal attention mechanism. The spherical upsampling and downsampling module realizes splitting and merging of spherical graph blocks based on a recursive process of the subdivision icosahedron method in the spherical uniform symmetric grid surface generation module, thereby realizing upsampling and downsampling. The application analysis module integrates the spherical convolution and pooling module, the spherical Transformer module, and the spherical upsampling and downsampling module, establishes a spherical autoencoder model, a segmentation model, and a classification model, and is used for target recognition, risk prediction, or anomaly detection tasks. The spherical uniform symmetric grid surface generation module specifically performs the following steps: based on an S0 curve, deforming the curve topology into a unit sphere in a standard space, and keeping the relative distance between the curve vertices and the geometric shape of the triangular facets within an error range that does not exceed a predetermined threshold during the deformation process. The spherical surface is deformed to a standard space by using surface registration and scaled to a unit sphere, i.e. a standard sphere S 2 The unit sphere is resampled to a uniform and highly symmetric graph structure by using the subdivision icosahedron method. The subdivision icosahedron method is an algorithm for approximating a sphere by recursively geometrically dividing an icosahedron, which can obtain uniformly distributed and symmetric vertices on the sphere. The icosahedron is a highly symmetric Platonic polyhedron composed of 20 equilateral triangular facets, 12 vertices, and 30 edges. The 12 vertices are located on the circumscribed sphere of the icosahedron and consist of three groups of symmetric points: (0,±1,±φ),(±1,±φ,0),(±φ,0,±1), where φ is the golden ratio, All vertices after normalization lie on the standard sphere S 2 ; The first-order icosahedron subdivision method and the second-order icosahedron subdivision method are used to reconstruct a uniform symmetric network on a standard sphere: The first-order icosahedron subdivision method includes: taking the midpoint of all edges, connecting the newly added midpoints to all facets to divide the original equilateral triangular facets into 4 equilateral triangular facets, and then normalizing the coordinates of the newly added midpoints to make the newly added midpoints fall on the standard sphere; after a single first-order subdivision, the number of facets S increases to 4×S, the number of edges E increases to 2×E+3×S, and the number of points V increases to V+E. The second-order icosahedron subdivision method comprises: setting the edge length as a, taking a point at a distance dist=2×sin(arcsin(a / 2) / 3) from an existing vertex, and taking the center point of an equilateral triangle, dividing each equilateral triangle into nine small equilateral triangles, normalizing the coordinates of the new points so that the new points fall on the standard spherical surface; after a single second-order subdivision, the number of faces S increases to 9×S, the number of edges E increases to 3×E+9×S, and the number of points V increases to V+2×E+S; applying the first-order icosahedron subdivision method and the second-order icosahedron subdivision method more than twice or alternately applying the first-order icosahedron subdivision method and the second-order icosahedron subdivision method more than twice, and setting the last time to apply the second-order icosahedron subdivision method; After obtaining the uniform and symmetric grid of the spherical surface, the natural neighborhood interpolation or the radial basis function interpolation method is applied to the interpolation of the features extracted by the image acquisition and preprocessing module; The natural neighbor interpolation includes: a set of original sampling points on a spherical surface Calculating a spherical Voronoi diagram, a Voronoi cell V of an i-th original sampling point p i i Containing a nearest vertex on the spherical surface from p i i is 1-n, Where x represents any one vertex on the spherical surface;​ d g denotes the geodesic distance; add the spherical vertex q obtained by the subdivision of regular icosahedron to the point set P and recalculate the Voronoi diagram, and identify the Voronoi cell V of the spherical vertex q q ; all the original vertices satisfying are called the natural neighborhood of the vertex q, and the contribution weight of the natural neighborhood vertices to the vertex q is calculated where Area represents the area; the eigenvalue f(q) of the vertex q is obtained by weighted summation calculation of the eigenvalues f(p i ) of the original sampling points p i in the natural neighborhood of the vertex q, f(q) =∑w i (q)f(p i ), wherein the eigenvalue f(p i ) represents the local physical or physiological attribute of the position where the original sampling point p i is located, and has been determined before interpolation. The radial basis function interpolation method comprises: for any spherical vertex q obtained by icosahedron subdivision method, taking the original sampling points p in the neighborhood of the vertex q i , so that d g i (q, p i )≤2σ, wherein σ is the standard deviation of the Gaussian kernel function; calculating the weight based on the Gaussian kernel function , wherein exp represents the natural exponential function; the eigenvalue f(q) of the vertex q is the weighted average of the eigenvalues f(p i ) of the original sampling points p in the neighborhood i wherein 2. The system of claim 1, wherein, The image acquisition and preprocessing module specifically performs the following steps: segmenting the original data to extract the region of interest, extracting the surface curve S0 based on the moving cube algorithm, and using the level set algorithm to expand S0 inward or outward to obtain the vertex one-to-one corresponding curve S1, and for brain data, extracting the vertex one-to-one corresponding gray matter inner surface curve S0 and outer surface curve S1, taking the corresponding vertex spacing as the curve thickness, and respectively reconstructing the intermediate curves S 0.25 、 0.5 S 0.75 25%, 50%, 75% spacing; the original data is three-dimensional medical image data reflecting the anatomical structure and functional information of the nervous system; For the same research object, a set of neural image data with curved surface morphological characteristics is obtained within a set time window by imaging technology, the set of neural image data includes multi-source image data, six-degree-of-freedom rigid registration is performed on the multi-source image data, and mutual information or minimum mean square error is used for alignment to a reference space, features are extracted at each curved surface node, including: geometric features of thickness, depth, and curvature are extracted at the S0 curved surface, and the signal intensity of the T1 weighted imaging T1WI and the fluid attenuated inversion recovery image FLAIR is mapped to the curved surface vertex; the T1 weighted imaging T1WI and the fluid attenuated inversion recovery image FLAIR are normalized respectively with the curved surface average signal value as the reference to obtain the normalized intensity features; for the brain diffusion tensor imaging DTI, a truncated axisymmetric Gaussian distribution is established respectively with each vertex on the S0 curved surface as the center point and the direction of the connecting line from the corresponding point on the S1 curved surface to the vertex on the S0 curved surface as the axis of symmetry, the intersection of the truncated axisymmetric Gaussian distribution region and the white matter is taken, and the weighted average fractional anisotropy FA, the average diffusion rate MD, the axial diffusion rate AD, and the radial diffusion rate RD are calculated; for the positron emission tomography image PET, the standardized uptake value ratio SUVR and the asymmetry index AI of each vertex on the S0-S1 curved surface are sampled.

3. The system of claim 2, wherein, The spherical convolution and pooling module specifically performs the following steps: Let the spherical distance between adjacent vertices be u, and the spherical convolution integrates the vertex information in the k×u neighborhood of the current vertex q, where k×u is the radius of the spherical convolution kernel; The vertex q is spaced from the current vertex u, The vertices of 6 and 2u are evenly distributed on 3 concentric circles; the spherical convolution kernel is distributed in the clockwise direction based on the meridian line passing through the current vertex q, and the vertices with a distance of k×u from the 12 vertices of the regular icosahedron are 5, and the average of the 5 vertices is taken as the filling. Based on the recursive process of the subdivided icosahedron, a spherical pooling method is established: the first-order subdivided icosahedron corresponds to a pooling method comprising: setting the current subdivision recursive number as l, then the center vertex and the outer hexagon vertex are the spherical vertices after l-1 recursive, and the remaining vertices are the newly generated vertices in this recursive; the spherical pooling integrates the information of the center vertex and all the first-order adjacent vertices of the center vertex into the center point, and removes the newly generated vertices in this recursive; The pooling method corresponding to the second-order subdivision method comprises: integrating the information of the newly generated vertices adjacent to the center point and the center points of the surrounding equilateral triangles into the center point.

4. The system of claim 3, wherein, The spherical Transformer module specifically performs the following steps: Icosahedral graph block partitioning: taking the vertex of the second-to-last icosahedral subdivision method as the center point, the adjacent second-order subdivision method vertices together constitute a graph block; in addition to the 12 vertices of the icosahedron, each center point has 12 adjacent vertices, of which 6 are located at the adjacent vertices of the equilateral triangle center and are shared with the surrounding graph blocks to capture the context information between adjacent blocks; the icosahedral vertex has only 10 adjacent vertices, including 5 edge center points and 5 equilateral triangle face center points; the mean value of the edge and equilateral triangle face center points is taken to fill the vector; after the spherical graph block is flattened, linear projection is performed, and each graph block is mapped to a feature matrix F patch : wherein denotes the real field, d is the vertex feature dimension, each spherical surface graph block has 13 vertices; The spherical Fourier position is used to positionally encode the spherical image block: where θ, φ are the spherical polar coordinates zenith and azimuth of the vertex, respectively, W m is a learnable parameter matrix; PE denotes position encoding; m denotes an index variable for summation; denotes the tensor product; A spherical Transformer model is established: for the feature vector h of the spherical image block, an attention mechanism is applied: wherein Attention represents the attention mechanism, Q, K and V are respectively the query vector, the key vector and the value vector; T represents matrix transposition; the spherical Transformer model performs multiplication on the attention and the feature vector h, then performs layer normalization, inputs a feedforward layer for integration, and then performs layer normalization again; establishing a spherical cross-modal Transformer model: the spherical cross-modal Transformer model includes a multi-head attention module with D layers of cross-modal attention modules CM β→α for feature vectors h α of multi-source data a and feature vectors h β of multi-source data b, the cross-modal attention module CM β→α passes information in b into a: where the query vector Q of β is obtained β from the feature vector h β The key vector K of α is obtained α and the value vector V α from h α d k is the dimension of the key vector; and is the parameter to be trained; For the multi-head attention module with D-layer cross-modal attention module CM β→α , the h α is used in each layer with cross-layer connection to make the first cross-layer summation with the output of the cross-modal attention module CM β→α , then layer normalization, input into the feedforward layer for nonlinear deformation, and the second cross-layer summation and layer normalization, wherein the cross-layer summation refers to the element-wise addition operation of the input feature vector h α of the current layer and the output feature vector h α processed by the cross-modal attention module.

5. The system of claim 4, characterized in that the spherical up-sampling and down-sampling module comprises a down-sampling layer and an up-sampling layer; The down-sampling layer uses a recursive process based on the method of subdividing the regular icosahedron to merge the spherical image blocks, the center point of the initial state image block is the vertex obtained by the last first-order subdivision, the feature vector mean of all vertices in the image block is taken and converged at the midpoint of the image block, and the weighted average of the feature of all vertices in the block is taken, and the weight is determined by the area proportion of the spherical Voronoi cell: wherein, A i Voronoi region area for vertex i; A total denotes the sum of all Voronoi region areas within a graph tile; f i denotes the feature vector of the i-th vertex; f pool denotes the output feature vector obtained after weighted pooling as the aggregated representation of the image tile; N denotes the total number of vertices; The feature mean of the vertex i and the adjacent newly sampled vertex of the current level is taken as the convergence target, and the down-sampling process reversely simulates the subdivision recursive step.

6. The system of claim 5, wherein, The up-sampling layer uses a recursive process based on the method of subdividing the regular icosahedron to reconstruct the vertex, starting from the vertex of the last regular icosahedron subdivision, backtracking to the initial regular icosahedron structure layer by layer, and the parent vertex of each level is determined by the subdivision tree topology; the low-resolution feature is restored to the high-resolution grid by linear interpolation: Perform first order feature interpolation: for each triangle edge, the two endpoints correspond to two original vertices A and B, the feature vectors of the two original vertices A and B are f A and f B , take the average of f A and f B , generate the edge midpoint feature f mid : The first-order feature interpolation corresponds to the projection of the midpoint of the set of regular icosahedron first-order subdivisions; the set of regular icosahedron first-order subdivisions refers to the set of two or more level grid structures generated by recursively subdividing the initial regular icosahedron, and each level corresponds to the vertex and face topology formed by one subdivision operation; Perform second order feature expansion: insert two new vertices at the trisection point of each edge, compute the feature f of the two new vertices by linear interpolation 1 / 3 and f 2 / 3 : Then, the f 1 / 3 and f 2 / 3 i.e. the feature mean of the six edge vertices as the face center point feature.

7. The system of claim 6, wherein, The application analysis module is used in the following application scenarios: Spherical autoencoder extracts depth features: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, and the spherical uniform symmetric grid surface generation module is used to establish a spherical grid, then the spherical Transformer module and the down-sampling layer are used to establish an encoder, and the spherical convolution, spherical pooling and up-sampling layer are used to establish a lightweight decoder, the mean square error is used as the loss function, and the output of the encoder is the depth feature of the data; A segmentation model that fuses multi-source data: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, the spherical uniform symmetric grid surface generation module is used to establish a spherical grid, the spherical convolution and pooling module is used to establish a spherical encoder, and the depth spherical features of multi-source data are extracted respectively, the spherical Transformer module is used to fuse the features, the spherical convolution and pooling module, the spherical up-sampling and down-sampling module are used to establish a U-Net network to realize data segmentation; The classification model of fused multi-source data: the image acquisition and preprocessing module is used to establish a curved surface and extract curved surface features, a spherical uniform symmetrical grid surface generation module is used to establish a spherical grid, a spherical convolution and pooling module is used to establish a spherical encoder, and the deep spherical features of multi-source data are extracted respectively, a spherical Transformer module is used to fuse the features, and after splicing, a ResNet network is established by using the spherical convolution and pooling module to realize data classification.

8. An electronic device, comprising: A processor and a memory are included, the memory stores program codes, and when the program codes are executed by the processor, the processor executes the system as claimed in any one of claims 1 to 7.

9. A storage medium, characterized by A computer program or instructions are stored, and when the computer program or instructions run on a computer, the system as claimed in any one of claims 1 to 7 is executed.

Images