Multi-angle photography and three-dimensional reconstruction method and system for soft tissue

Abstract

The present invention provides a multi-angle photography and three-dimensional reconstruction method and system for soft tissue, which relates to the technical field of soft tissue three-dimensional reconstruction, including analyzing soft tissue features based on deep learning, generating tube motion trajectory and sampling interval data, controlling a mechanical arm to drive the tube to collect projection images at multiple angles; using an image quality assessment network to screen the optimal image sequence, and performing preprocessing such as material decomposition, feature extraction, spatial alignment, and image optimization; inputting the optimized image and material features into a reconstruction model, performing iterative calculation in combination with an anatomical structure template to generate density distribution data, and performing surface reconstruction and intelligent tissue segmentation based on graph convolution based on the density distribution data, and finally generating a three-dimensional model containing tissue stratification information.

Description

Technical Field

[0001] The present invention relates to the technical field of soft tissue three-dimensional reconstruction, and in particular to a multi-angle photography and three-dimensional reconstruction method and system for soft tissue. Background Art

[0002] The three-dimensional reconstruction technology of soft tissue has important application value in the field of medical imaging. By collecting multi-angle projection images and combining computer vision technology, an accurate three-dimensional representation of the internal structure of soft tissue can be obtained. Traditional soft tissue imaging methods mainly rely on technologies such as X-ray computed tomography and magnetic resonance imaging. These technologies obtain tissue density and morphological information through the interaction of radiation or magnetic field signals of different energy levels with human tissue. With the development of deep learning technology, feature extraction and image reconstruction methods based on neural networks have made significant progress in improving the quality of soft tissue imaging.

[0003] However, there are still several problems with the soft tissue imaging and reconstruction technology in the existing technology. The traditional fixed-trajectory scanning method is difficult to adaptively adjust according to the characteristics of different soft tissues, resulting in low sampling efficiency and uneven image quality. In the process of multi-angle image acquisition, due to the influence of factors such as patient respiratory movement and equipment vibration, spatial position deviation and motion artifacts are easily generated, affecting the reconstruction accuracy. The biomechanical properties and deformation laws of different tissue levels are not fully considered, making it difficult to accurately reflect the hierarchical structural characteristics of soft tissues. The performance is limited in noise suppression and contrast enhancement, and there is a lack of accurate analysis capabilities for material properties.

[0004] In summary, there is an urgent need for an adaptive multi-angle photography and three-dimensional reconstruction method. Through deep convolutional neural network analysis of soft tissue features, intelligent optimization of the tube motion trajectory is achieved; visual servo control technology is used to ensure accurate tracking of the acquisition equipment; high-precision three-dimensional reconstruction is achieved by combining compressed sensing reconstruction and biomechanical constraints; the reconstruction results are intelligently segmented through tissue classification models, and finally a three-dimensional model with an accurate hierarchical structure is obtained. The present invention can solve the problems in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a multi-angle photography and three-dimensional reconstruction method and system for soft tissue, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] A multi-angle photography and three-dimensional reconstruction method for soft tissue is provided, comprising:

[0008] A deep convolutional neural network is used to perform feature analysis on the soft tissue under examination to obtain volume parameters and density distribution parameters of the soft tissue; based on the volume parameters and density distribution parameters, a trajectory optimization algorithm is used to generate tube motion trajectory data and sampling interval data; a multi-degree-of-freedom manipulator is controlled to drive the tube to move according to the tube motion trajectory data, and a visual servo controller is used to adjust the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking; according to the sampling interval data, the tube is controlled to collect soft tissue projection images in a preset first energy level range and a second energy level range, respectively, to obtain a multi-angle soft tissue projection data set;

[0009] Inputting the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screening to obtain optimal image sequence data according to the image quality score data; performing material decomposition processing on the optimal image sequence data, and extracting soft tissue structure feature data and material feature data using a multi-scale feature fusion network; performing feature point matching on the soft tissue structure feature data and inertial sensor data, and generating registered image data through a self-correcting spatial registration algorithm; and performing adaptive noise reduction processing and local contrast enhancement processing based on wavelet transform on the registered image data in sequence to obtain optimized multi-angle image data;

[0010] The optimized multi-angle image data and the material feature data are input into a compressed sensing reconstruction model, and regularized constraints are constructed in combination with a human soft tissue anatomical structure template, and soft tissue density distribution data is generated through iterative calculation; based on the soft tissue density distribution data, an adaptive meshing algorithm is used to perform surface reconstruction to generate initial three-dimensional model data; the initial three-dimensional model data is input into a graph convolutional feature extraction network to extract topological feature data and morphological feature data; a tissue classification model is constructed based on the topological feature data and the morphological feature data, and different density areas are intelligently segmented to generate three-dimensional model data containing tissue stratification information.

[0011] In an optional embodiment,

[0012] Based on volume parameters and density distribution parameters, the trajectory optimization algorithm generates tube motion trajectory data and sampling interval data including:

[0013] Determine a spatial envelope region based on the volume parameter, establish a three-dimensional grid in the spatial envelope region, recursively refine the three-dimensional grid using an adaptive grid division strategy, determine the degree of grid refinement by calculating a density curvature value and a gradient change rate of the grid unit, subdivide the grid unit when the density curvature value is greater than a first preset threshold or the gradient change rate is greater than a second preset threshold, and merge the grid units when both the density curvature value and the gradient change rate are less than a third preset threshold;

[0014] Constructing a density feature tensor, wherein the density feature tensor includes a density value, a density gradient value, and a density Hessian matrix eigenvalue of a grid node, calculating an importance score of the grid node based on the density feature tensor, selecting grid nodes whose importance scores are greater than a preset threshold as candidate sampling nodes, performing hierarchical clustering on the candidate sampling nodes, and introducing a topological constraint in each cluster to determine a key sampling point;

[0015] Establish an imaging quality evaluation model that combines ray attenuation and scattering effects, add the image clarity index and signal-to-noise ratio index output by the imaging quality evaluation model to the path optimization objective function, construct an optimization problem of the optimal access sequence based on the key sampling points, and use an iterative optimization algorithm with an adaptive step size to solve the optimal path connecting the key sampling points;

[0016] A local search strategy is introduced in the optimization process. When a local suboptimal solution is detected in the connection path between the key sampling points, the local path segment is reconstructed based on the taboo search method, and the reconstructed path is corrected using the path smoothness constraint to generate the ball tube motion trajectory data connecting all the key sampling points.

[0017] The path between adjacent key sampling points is divided into a plurality of sampling areas according to the tube motion trajectory data, a density gradient histogram is established in each sampling area, an initial value of the sampling interval is determined based on the statistical characteristics of the density gradient histogram, and the sampling interval is adaptively adjusted in combination with the degree of tissue interface transition of the key sampling points to generate the sampling interval data.

[0018] In an optional embodiment,

[0019] Controlling the multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjusting the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller includes:

[0020] Constructing a mapping relationship between the visual feature space and the control space between the tube and the detector assembly, extracting feature points and feature point coordinates of the tube and the detector assembly, establishing a three-dimensional coordinate mapping of the feature points in the visual system coordinate system, calculating the partial derivatives of the image features with respect to the motion of the multi-degree-of-freedom manipulator through the three-dimensional coordinate mapping, and generating an image Jacobian matrix;

[0021] Collecting the real-time position coordinates and attitude angle of the detector assembly relative to the tube, calculating the position deviation vector and attitude deviation vector of the detector assembly relative to the tube, combining the position deviation vector and the attitude deviation vector to construct an error vector, determining a dynamic compensation control law based on the error vector and the image Jacobian matrix, and generating an initial control output of the multi-degree-of-freedom manipulator;

[0022] Based on the comparison result of the modulus value of the error vector and the preset error threshold, the adjustment direction of the proportional gain coefficient is determined, and according to the preset motion stability index of the multi-degree-of-freedom manipulator, the adjustment direction of the damping coefficient is determined, when the modulus value of the error vector is greater than the first error threshold, the proportional gain coefficient is reduced according to the preset proportion, and when the modulus value of the error vector is less than the second error threshold, the proportional gain coefficient is increased according to the preset proportion, and the control parameters after adaptive adjustment are generated;

[0023] In a preset observation time window, the relative feature change between the tube and the detector assembly and the joint motion increment of the multi-degree-of-freedom manipulator are collected, the relative feature change and the joint motion increment are input into a recursive least squares estimator to calculate a covariance matrix, a Kalman gain matrix is ​​calculated based on the covariance matrix, and the image Jacobian matrix is ​​updated based on the Kalman gain matrix to obtain a real-time image Jacobian matrix;

[0024] The initial control output and the real-time image Jacobian matrix are input into an auto-disturbance rejection observer, the external comprehensive interference term in the motion process of the multi-degree-of-freedom manipulator is calculated, and the compensation value of the external comprehensive interference term is superimposed on the initial control output to generate a compensation control output; the deviation distance between the actual tracking trajectory of the detector assembly relative to the tube and the expected tracking trajectory is calculated, an adaptive sliding mode control term is constructed according to the deviation distance, and the adaptive sliding mode control term is superimposed on the compensation control output to generate a sliding mode correction control output;

[0025] The operation is repeatedly performed in a plurality of sampling cycles to generate a final control instruction for adjusting the multi-degree-of-freedom manipulator.

[0026] In an optional embodiment,

[0027] Inputting the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screening and obtaining optimal image sequence data according to the image quality score data includes:

[0028] Input the multi-angle soft tissue projection data into the feature separation network, extract the detail features and structural features of the image respectively through the dual-branch structure, construct the frequency domain attention map respectively, enhance the detail features and structural features according to the frequency domain attention map, and obtain enhanced dual frequency domain feature data;

[0029] The dual-frequency domain feature data is input into a residual dense connection network including a plurality of residual dense blocks, a plurality of convolutional layers are connected in series in a dense connection manner in each residual dense block, and features are reused between adjacent residual dense blocks through skip connections to extract multi-level image features and obtain a multi-scale feature map;

[0030] Constructing a self-attention pyramid module, dividing the multi-scale feature map into feature blocks of different scales, calculating the correlation matrix between different feature blocks, generating attention weights according to the correlation matrix, adaptively weighting the feature blocks, and obtaining fused feature data;

[0031] Inputting the fused feature data into an uncertainty learning module, wherein the uncertainty learning module comprises two parallel branches, outputting a quality score prediction value through a first branch, outputting a prediction uncertainty estimate value through a second branch, combining the quality score prediction value and the prediction uncertainty estimate value to generate a credibility-weighted quality score;

[0032] Sort the images according to the credibility-weighted quality scores to obtain an initial candidate sequence, perform pairwise similarity calculation on the images in the initial candidate sequence, and construct a similarity matrix; perform cluster analysis on the initial candidate sequence based on the similarity matrix, classify images with similarities higher than a preset similarity threshold into the same category, and select the image with the highest quality score in each category as a category labeled image;

[0033] The mutual information between each image in the initial candidate sequence and the corresponding category labeled image is calculated, and the images are reordered according to a weighted combination of the mutual information and the quality score. According to a preset front row ratio, images with corresponding rankings and meeting the minimum image quantity requirement are selected to form an optimal image sequence.

[0034] In an optional embodiment,

[0035] Matching the soft tissue structure feature data with the inertial sensor data at feature points, and generating registration image data by a self-correction spatial registration algorithm includes:

[0036] In the soft tissue structural feature data, a circular neighborhood with a preset radius is constructed for each candidate feature point, and the grayscale difference between the pixel point in the circular neighborhood and the center point and the grayscale variance of the circular neighborhood are calculated. When the grayscale difference of a plurality of consecutive pixel points is greater than a preset grayscale difference threshold, and the grayscale variance is greater than a preset grayscale variance threshold, the candidate feature point is marked as a structural feature point, and a soft tissue feature point set is obtained;

[0037] Performing Kalman filter preprocessing on the inertial sensor data to obtain filtered data, performing spatial integration on the filtered data to obtain sensor position coordinates, and projecting them onto an image plane to obtain a position feature point set; calculating the Mahalanobis distance for each position feature point in the position feature point set, eliminating position feature points whose Mahalanobis distance is greater than a preset distance threshold, and obtaining an optimized position feature point set;

[0038] Calculating the matrix eigenvalue ratio of the neighborhood corresponding to each feature point in the soft tissue feature point set and the optimized position feature point set to obtain the local structure complexity;

[0039] Adaptively adjusting the sampling radius according to the local structure complexity, constructing a multi-scale sampling pattern around each feature point, and extracting feature description vectors of each feature point in the soft tissue feature point set and the optimized position feature point set based on the multi-scale sampling pattern;

[0040] Calculating the mutual information coefficient and the relative position relationship parameter between the feature description vectors, matching the soft tissue feature point set and the optimized position feature point set according to the mutual information coefficient and the relative position relationship parameter, and obtaining an initial matching point pair set;

[0041] Calculate the mutual information coefficient of the feature description vector and the consistency degree of local space transformation of each pair of matching points in the initial matching point pair set, and generate a matching credibility score in combination with the motion constraints provided by the inertial sensor data; screen the initial matching point pair set according to the matching credibility score, and select the matching point pairs whose matching credibility scores are higher than a preset matching credibility score threshold as the preferred matching point pairs; construct a spatial transformation matrix based on the preferred matching point pairs, and adaptively determine the loss function threshold parameter according to the current matching error distribution;

[0042] Based on the loss function threshold parameter, combined with the Huber loss function, the parameters of the spatial transformation matrix are optimized, and when the parameter change of the spatial transformation matrix during the optimization process is less than a preset matrix parameter change threshold, the current spatial transformation matrix is ​​determined as the optimal transformation matrix;

[0043] The soft tissue structure feature data is spatially transformed according to the optimal transformation matrix to generate registration image data.

[0044] In an optional embodiment,

[0045] Inputting the optimized multi-angle image data and the material characteristic data into the compressed sensing reconstruction model, building regularized constraint conditions in combination with the human soft tissue anatomical structure template, and generating soft tissue density distribution data through iterative calculation includes:

[0046] Obtaining photon attenuation coefficients according to material characteristic data, constructing a tissue attenuation coefficient matrix, establishing a linear density mapping function based on the tissue attenuation coefficient matrix, and obtaining a material-density mapping model;

[0047] Constructing an orthogonal sparse basis matrix based on the optimized multi-angle image data, and calculating the sparse representation coefficients of the projection data under the orthogonal sparse basis matrix;

[0048] Extracting tissue interface position coordinates and interface curvature values ​​from a human soft tissue anatomical structure template, and constructing anatomical feature constraints based on the interface position coordinates and the interface curvature values;

[0049] Using the interface position coordinates, the density reconstruction space is divided into a plurality of continuous sub-regions, a density continuity constraint is imposed inside the continuous sub-regions, and a piecewise continuous constraint function with controllable jumps is constructed at the interface position coordinates to generate a density distribution regularization term;

[0050] Calculating structural similarity coefficients of adjacent image blocks in the reconstruction space, constructing a non-local difference operator according to the structural similarity coefficient, applying the non-local difference operator to density distribution data, and establishing a structure preservation constraint term;

[0051] The material-density mapping model, the anatomical feature constraint, the density distribution regularization term and the structure preservation constraint term are combined into a joint optimization objective function; the joint optimization objective function is decomposed into variables by an augmented Lagrangian method to obtain a projection consistency constraint term and a structure preservation constraint term;

[0052] Based on the projection consistency constraint, the projection coefficient matrix is ​​solved by the conjugate gradient method, and based on the structure preservation constraint, the constraint parameters are optimized by the proximal gradient descent method to obtain the density distribution image of the current iteration;

[0053] A three-layer dictionary learning network is constructed based on historical reconstruction data, and feature dictionary groups with different scale organizational structures are obtained through training.

[0054] Sparsely decompose the density distribution image of the current iteration in the feature dictionary group, calculate the reconstruction error at each scale, select the feature dictionary with the smallest reconstruction error to reconstruct the density distribution, and obtain an optimized density image;

[0055] The Euclidean distance difference between the optimized density images obtained in two adjacent iterations is calculated, and when the Euclidean distance difference is less than a preset density image distance difference threshold, the current optimized density image is determined as the soft tissue density distribution data.

[0056] In an optional embodiment,

[0057] Constructing a tissue classification model based on the topological feature data and the morphological feature data, intelligently segmenting different density areas, and generating three-dimensional model data containing tissue layering information includes:

[0058] Inputting the topological feature data and the morphological feature data into a three-dimensional convolution extraction network to obtain a multi-scale feature mapping sequence, constructing a feature hierarchy network based on the multi-scale feature mapping sequence, calculating a local structure tensor matrix, extracting a tissue boundary direction vector and a principal curvature parameter from the local structure tensor matrix, and generating a structure description vector;

[0059] Using the structure description vector to construct an organizational connectivity graph model, wherein the graph nodes of the organizational connectivity graph model store the local structure description vector, and the graph edges store the spatial position relationship between adjacent nodes;

[0060] Constructing a feature enhancement unit based on phase change diffusion, inputting the tissue connectivity graph model into a nonlinear diffusion equation, solving the nonlinear diffusion equation, obtaining enhanced boundary features and enhanced internal structure features, and updating node features of the tissue connectivity graph model;

[0061] Establish a soft tissue elastic mechanics model based on domain knowledge, convert the tissue stress-strain relationship into node displacement constraint parameters, generate biomechanical constraint conditions, calculate tissue deformation strain energy according to the biomechanical constraint conditions, and construct a regional growth function based on the tissue deformation strain energy minimization criterion;

[0062] The tissue connectivity graph model is segmented iteratively and hierarchically according to the regional growth function to obtain segmented regions, the values ​​of tissue deformation strain energy of the segmented regions are calculated, and the boundaries where the difference in tissue deformation strain energy of adjacent regions is greater than a preset strain energy difference threshold are determined as tissue demarcation point sets; a tissue hierarchy label set is constructed based on the tissue demarcation point set, a tissue hierarchy rule base is established, and tissue hierarchy data is generated;

[0063] Performing credibility calculation on the organizational hierarchy data to obtain a regional classification credibility value, and marking the regions whose regional classification credibility values ​​are lower than a preset credibility threshold as a set of regions to be optimized;

[0064] Adopting a neighborhood information transfer algorithm to iteratively optimize the set of regions to be optimized, updating node feature parameters and edge constraint parameters, until the classification credibility values ​​of all regions are greater than the preset credibility threshold, and obtaining optimized classification identification data;

[0065] The optimized classification identification data and the density distribution data are spatially registered and fused to generate a three-dimensional soft tissue model with anatomical structure annotations.

[0066] According to a second aspect of the embodiments of the present invention,

[0067] A multi-angle photography and three-dimensional reconstruction system for soft tissue is provided, comprising:

[0068] The first unit is used to perform feature analysis on the inspected soft tissue using a deep convolutional neural network to obtain volume parameters and density distribution parameters of the soft tissue; based on the volume parameters and density distribution parameters, generate tube motion trajectory data and sampling interval data through a trajectory optimization algorithm; control a multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjust the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller; control the tube to collect soft tissue projection images in a preset first energy level range and a second energy level range respectively according to the sampling interval data, and obtain a multi-angle soft tissue projection data set;

[0069] The second unit is used to input the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screen and obtain optimal image sequence data according to the image quality score data; perform material decomposition processing on the optimal image sequence data, and use a multi-scale feature fusion network to extract soft tissue structure feature data and material feature data; perform feature point matching on the soft tissue structure feature data and inertial sensor data, and generate registration image data through a self-correction spatial registration algorithm; and perform adaptive noise reduction processing and local contrast enhancement processing based on wavelet transform on the registration image data in sequence to obtain optimized multi-angle image data;

[0070] The third unit is used to input the optimized multi-angle image data and the material feature data into a compressed sensing reconstruction model, build regularized constraints in combination with a human soft tissue anatomical structure template, and generate soft tissue density distribution data through iterative calculation; based on the soft tissue density distribution data, an adaptive meshing algorithm is used to perform surface reconstruction to generate initial three-dimensional model data; the initial three-dimensional model data is input into a graph convolutional feature extraction network to extract topological feature data and morphological feature data; a tissue classification model is built based on the topological feature data and the morphological feature data, different density areas are intelligently segmented, and three-dimensional model data containing tissue stratification information is generated.

[0071] According to a third aspect of the embodiments of the present invention,

[0072] An electronic device is provided, comprising:

[0073] processor;

[0074] a memory for storing processor-executable instructions;

[0075] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0076] According to a fourth aspect of the embodiments of the present invention,

[0077] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0078] In an embodiment of the present invention, image quality assessment and screening are performed through a deep learning network, and image data is optimized in combination with processing methods such as adaptive noise reduction and local contrast enhancement, thereby effectively improving the quality of the projected image; a multi-scale feature fusion network is used to extract the structure and material characteristics of soft tissue, and regularization constraints are constructed in combination with a human anatomical structure template, thereby improving the precision and accuracy of three-dimensional reconstruction; a graph convolution feature extraction network is used to extract the topological and morphological characteristics of the three-dimensional model, and a tissue classification model is constructed, thereby realizing intelligent segmentation of areas of different densities, and finally generating a three-dimensional model containing tissue stratification information, which is conducive to a more detailed analysis of the structure and pathological conditions of soft tissue; a deep learning network is used to analyze soft tissue characteristics, and a trajectory optimization algorithm is used to generate the tube motion trajectory and sampling interval data, thereby realizing synchronous tracking of the tube and the detector, with a high degree of automation, optimizing the imaging process, and improving imaging efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a flowchart of a multi-angle photography and three-dimensional reconstruction method for soft tissue according to an embodiment of the present invention;

[0080] Figure 2 It is a schematic structural diagram of a multi-angle photography and three-dimensional reconstruction system for soft tissue according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0083] Figure 1 FIG. 1 is a flow chart of a multi-angle photography and three-dimensional reconstruction method for soft tissue according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0084] S101. Use a deep convolutional neural network to perform feature analysis on the soft tissue under examination to obtain volume parameters and density distribution parameters of the soft tissue; based on the volume parameters and density distribution parameters, generate tube motion trajectory data and sampling interval data through a trajectory optimization algorithm; control a multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjust the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller; control the tube to collect soft tissue projection images in a preset first energy level range and a second energy level range respectively according to the sampling interval data, and obtain a multi-angle soft tissue projection data set;

[0085] The first energy level range specifically refers to a lower X-ray energy range, usually a low kilovolt value, such as 40-70 kVp), which is used to capture soft tissue details and data that is more sensitive to attenuation information of low-density materials. In the corresponding energy range, the penetration ability of X-rays is weaker, but the contrast of soft tissue and low-density areas is higher.

[0086] The second energy level range specifically refers to a higher X-ray energy range, usually medium to high kilovolt values, such as 80-140kVp), which is used to penetrate higher density tissues (such as bones) or for deep comparative analysis of soft tissue and material differences. In the corresponding energy range, X-rays have stronger penetrating power, which can reduce the scattering effect in low energy rays and provide supplementary information for multi-energy imaging.

[0087] A deep convolutional neural network is used to perform feature analysis on the soft tissue under examination. The network consists of a feature extraction layer and an analysis layer. Through multi-layer convolution operations, the density distribution, boundary contours, and spatial structure of the soft tissue are extracted. After network analysis, two types of key parameters are output: volume parameters that reflect the overall size and shape of the soft tissue, and density distribution parameters that characterize the density changes of the internal structure of the soft tissue.

[0088] Based on the acquired volume parameters and density distribution parameters, the trajectory optimization algorithm is used for calculation and analysis. The algorithm comprehensively considers the spatial distribution characteristics of soft tissues, scanning coverage requirements, and equipment motion constraints, and optimizes and generates two sets of data: trajectory data for guiding the spatial motion of the tube, and sampling interval data for determining the distribution of sampling positions. The trajectory data contains the spatial coordinate sequence of the tube motion, and the sampling interval data contains the position and sampling timing information of each sampling point.

[0089] The multi-degree-of-freedom manipulator control system is used to execute the tube movement. The manipulator drives the tube to move along the predetermined path according to the trajectory data. At the same time, it is equipped with a visual servo controller to obtain the tube position information in real time, and adjust the multi-degree-of-freedom manipulator to drive the detector assembly to track the position accordingly, ensuring that the tube and the detector maintain an accurate relative position relationship and realizing synchronous motion control.

[0090] The specific sampling time and position are determined according to the sampling interval data, and two preset energy ranges are used for projection imaging at each sampling position. The ray energies in the first energy level range and the second energy level range have different penetration capabilities, which can obtain structural information at different levels of soft tissue. Through this dual-energy acquisition method, projection images of soft tissue are obtained at multiple angles, and finally a multi-angle soft tissue projection data set containing rich structural information is formed.

[0091] In this embodiment, a deep convolutional neural network is used to extract the volume parameters and density distribution parameters of soft tissue to provide high-precision basic data for subsequent processing; the tube motion trajectory and sampling interval data are generated based on the trajectory optimization algorithm to improve scanning efficiency and data coverage and reduce redundant sampling; the multi-degree-of-freedom manipulator works in conjunction with the visual servo controller to ensure the motion synchronization of the tube and detector components and improve the stability and accuracy of image acquisition; multi-angle soft tissue projection images are acquired at different energy levels to enrich the data set and improve the contrast and detail resolution of soft tissue imaging; feature analysis, trajectory generation, equipment control and multi-energy data acquisition are integrated to achieve an automated and efficient soft tissue imaging process.

[0092] S102. Input the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and select and obtain optimal image sequence data according to the image quality score data; perform material decomposition processing on the optimal image sequence data, and use a multi-scale feature fusion network to extract soft tissue structure feature data and material feature data; perform feature point matching on the soft tissue structure feature data and inertial sensor data, and generate registered image data through a self-correction spatial registration algorithm; perform adaptive noise reduction processing based on wavelet transform and local contrast enhancement processing on the registered image data in sequence to obtain optimized multi-angle image data;

[0093] The collected multi-angle soft tissue projection data set is input into the image quality assessment network, which evaluates and scores the clarity, contrast, signal-to-noise ratio and other indicators of each projection image to generate quality score data. Based on the score data, the image sequence with the best quality is selected and the images with poor quality are removed.

[0094] The optimal image sequence after screening is subjected to material decomposition processing, and the image features are analyzed using a multi-scale feature fusion network. The network extracts and fuses features at multiple scale levels and outputs two types of data: structural feature data reflecting the morphological structure of the tissue, and material feature data representing the properties of the tissue material.

[0095] The extracted soft tissue structure feature data is matched with the inertial sensor data recorded during the acquisition process, and the image is spatially corrected through a self-correcting spatial registration algorithm to eliminate the spatial position deviation caused by equipment movement and patient movement, and generate registered image data.

[0096] The registered image data is optimized in two steps: first, an adaptive denoising algorithm based on wavelet transform is used to remove image noise, and then local contrast enhancement is performed to improve image detail performance, and finally optimized multi-angle image data is obtained.

[0097] In this embodiment, the optimal image sequence is screened through an image quality assessment network to ensure that subsequent processing is based on high-quality data and to improve the reliability of the results; a multi-scale feature fusion network is used to extract soft tissue structure and material feature data to enhance the recognition ability of complex tissue information; inertial sensor data is combined for feature point matching, and a self-correcting spatial registration algorithm is used to generate high-precision registration image data to reduce errors; adaptive noise reduction processing and local contrast enhancement technology based on wavelet transform are used to further optimize image clarity and detail performance; multi-angle image data are systematically processed and optimized to provide rich and accurate image information for subsequent analysis and diagnosis.

[0098] S103. Input the optimized multi-angle image data and the material feature data into a compressed sensing reconstruction model, build regularized constraints in combination with a human soft tissue anatomical structure template, and generate soft tissue density distribution data through iterative calculation; based on the soft tissue density distribution data, use an adaptive meshing algorithm to perform surface reconstruction to generate initial three-dimensional model data; input the initial three-dimensional model data into a graph convolutional feature extraction network to extract topological feature data and morphological feature data; build a tissue classification model based on the topological feature data and the morphological feature data, perform intelligent segmentation on different density areas, and generate three-dimensional model data containing tissue stratification information.

[0099] The optimized multi-angle image data and material feature data are input into the compressed sensing reconstruction model, and the human soft tissue anatomical structure template is introduced as prior knowledge to construct regularized constraints. Through iterative optimization calculation, the density distribution data representing the internal structure of the soft tissue is reconstructed.

[0100] Based on the acquired soft tissue density distribution data, an adaptive meshing algorithm is used for 3D reconstruction. The algorithm adaptively adjusts the mesh density according to the gradient change of density distribution, performs surface fitting on the soft tissue surface, and generates the initial 3D model data.

[0101] The initial 3D model data is input into the graph convolution feature extraction network, which analyzes the structural features of the 3D model through graph convolution operations and extracts two types of feature data: topological feature data describing the connection relationship of soft tissue, and morphological feature data describing the shape characteristics of soft tissue.

[0102] Based on the extracted topological and morphological features, a tissue classification model is constructed, which intelligently segments and classifies different density areas in the 3D model. Through the analysis of the classification model, the boundaries of different tissue layers are identified, and finally a 3D model data containing complete tissue layering information is generated.

[0103] In this embodiment, by optimizing multi-angle image data and material feature data and combining the regularization constraints of the anatomical template, a compressed sensing reconstruction model is used to generate high-precision soft tissue density distribution data; an adaptive meshing algorithm is used for surface reconstruction to quickly generate initial three-dimensional model data to support subsequent analysis and operations; with the help of a graph convolutional feature extraction network, topological features and morphological features are obtained to accurately describe tissue structure and morphological features; a tissue classification model is constructed to intelligently segment different density areas to generate three-dimensional model data with tissue stratification information, providing multi-level support for soft tissue analysis and diagnosis; and fine stratification and classification of different density areas in soft tissue are achieved to improve the pertinence and accuracy of diagnosis and treatment.

[0104] In an optional implementation, generating tube motion trajectory data and sampling interval data through a trajectory optimization algorithm based on volume parameters and density distribution parameters includes:

[0105] Determine a spatial envelope region based on the volume parameter, establish a three-dimensional grid in the spatial envelope region, recursively refine the three-dimensional grid using an adaptive grid division strategy, determine the degree of grid refinement by calculating a density curvature value and a gradient change rate of the grid unit, subdivide the grid unit when the density curvature value is greater than a first preset threshold or the gradient change rate is greater than a second preset threshold, and merge the grid units when both the density curvature value and the gradient change rate are less than a third preset threshold;

[0106] Constructing a density feature tensor, wherein the density feature tensor includes a density value, a density gradient value, and a density Hessian matrix eigenvalue of a grid node, calculating an importance score of the grid node based on the density feature tensor, selecting grid nodes whose importance scores are greater than a preset threshold as candidate sampling nodes, performing hierarchical clustering on the candidate sampling nodes, and introducing a topological constraint in each cluster to determine a key sampling point;

[0107] Establish an imaging quality evaluation model that combines ray attenuation and scattering effects, add the image clarity index and signal-to-noise ratio index output by the imaging quality evaluation model to the path optimization objective function, construct an optimization problem of the optimal access sequence based on the key sampling points, and use an iterative optimization algorithm with an adaptive step size to solve the optimal path connecting the key sampling points;

[0108] A local search strategy is introduced in the optimization process. When a local suboptimal solution is detected in the connection path between the key sampling points, the local path segment is reconstructed based on the taboo search method, and the reconstructed path is corrected using the path smoothness constraint to generate the ball tube motion trajectory data connecting all the key sampling points.

[0109] The path between adjacent key sampling points is divided into a plurality of sampling areas according to the tube motion trajectory data, a density gradient histogram is established in each sampling area, an initial value of the sampling interval is determined based on the statistical characteristics of the density gradient histogram, and the sampling interval is adaptively adjusted in combination with the degree of tissue interface transition of the key sampling points to generate the sampling interval data.

[0110] The density Hessian matrix eigenvalue specifically refers to a numerical indicator that describes the curvature characteristics of the density distribution, which is derived from the local change characteristics of the density distribution and indicates the degree and trend of density changes in different directions. Specifically, the eigenvalue reflects how the density expands or contracts in a certain direction, wherein a positive eigenvalue indicates that the change in density in this direction is expansion, indicating that the density value of the area gradually increases with the change in position; a negative eigenvalue indicates that the change in density in this direction is contraction, indicating that the density value of the area gradually decreases with the change in position; the size of the absolute value indicates the severity of the density change, the larger the absolute value, the more significant the density curvature in this direction, and the more drastic the change; the eigenvalue combination can determine the density distribution shape of the local area by comprehensively considering the eigenvalues ​​in multiple directions, such as whether it is a sharp edge, a smooth plane, or a feature point with significant curvature.

[0111] In a specific implementation, first, a spatial envelope region is determined according to a given volume parameter, such as the size range of a certain organ of the human body. An initial three-dimensional grid is established in this spatial region. Then, the three-dimensional grid is recursively refined using an adaptive grid partitioning strategy. Specifically, the density curvature value and density gradient change rate of each grid unit are calculated. The density curvature value reflects the curvature degree of the density distribution, and the gradient change rate reflects the drastic degree of density change. If the density curvature value of a grid unit is greater than a preset first threshold, or its density gradient change rate is greater than a preset second threshold, it is considered that the density change of the unit is more complex and needs to be further subdivided. Conversely, if the density curvature value and density gradient change rate of a grid unit are both less than a preset third threshold, it is considered that the density change of the unit is gentle, and it can be merged with adjacent units to reduce the amount of calculation. For example, assuming that the first threshold is 0.1, the second threshold is 0.05, and the third threshold is 0.01. For a grid unit with a density curvature value of 0.12, it needs to be subdivided because it is greater than the first threshold. This process is repeated until the density curvature values ​​and gradient change rates of all grid cells meet the preset conditions.

[0112] Next, a density feature tensor is constructed. For each grid node, its density value, density gradient value, and density Hessian matrix eigenvalue are calculated, and these values ​​are combined into a vector, namely the density feature tensor. The density value represents the material density of the point, the density gradient value represents the direction and magnitude of the density change at the point, and the density Hessian matrix eigenvalue represents the curvature of the density change at the point. Then, the importance score of each grid node is calculated based on the density feature tensor. For example, a higher importance score can be assigned according to the size of the density gradient, because areas with larger density gradients usually correspond to the boundaries of tissues, which are critical for imaging quality. Grid nodes with an importance score greater than a preset threshold are selected as candidate sampling nodes. For example, if the preset threshold is 0.8, nodes with an importance score greater than 0.8 are selected. These candidate sampling nodes are hierarchically clustered, and nodes that are spatially close and have similar density features are divided into the same cluster. In each cluster, topological constraints are introduced to determine the key sampling points. For example, the node with the largest density gradient in each cluster can be selected as the key sampling point.

[0113] Then, an imaging quality evaluation model combining ray attenuation and scattering effects is established. This model can simulate the propagation process of X-rays in human tissues and predict the quality of the generated images. The image clarity index and signal-to-noise ratio index output by the model are added to the path optimization objective function. An optimization problem of the optimal access sequence is constructed based on key sampling points. The goal is to find an optimal path connecting all key sampling points so that the value of the objective function is maximized. An iterative optimization algorithm with adaptive step size is used to solve the optimization problem. In the optimization process, a local search strategy is introduced. When a local suboptimal solution is detected in the connection path between key sampling points, the local path segment is reconstructed based on the taboo search method, and the reconstructed path is corrected using the path smoothness constraint. Finally, the tube motion trajectory data connecting all key sampling points is generated.

[0114] Finally, the path between adjacent key sampling points is divided into multiple sampling areas according to the tube motion trajectory data. A density gradient histogram is established in each sampling area to count the frequency of occurrence of different density gradient values ​​in the area. Based on the statistical characteristics of the density gradient histogram, the initial value of the sampling interval is determined. For example, in areas with large changes in density gradient, a smaller sampling interval should be used to capture finer details. Combined with the degree of tissue interface transition at the key sampling points, the sampling interval is adaptively adjusted. Finally, the sampling interval data is generated.

[0115] In this embodiment, through adaptive grid refinement and sampling point selection based on density features, the detailed information of the tissue can be captured more effectively, thereby improving the clarity and signal-to-noise ratio of the image; by optimizing the tube motion trajectory and sampling interval, unnecessary sampling can be reduced, thereby reducing the patient's radiation dose; the optimized sampling strategy can reduce the number of sampling times, thereby shortening the scanning time and improving the inspection efficiency.

[0116] In an optional embodiment, controlling a multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjusting the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller includes:

[0117] Constructing a mapping relationship between the visual feature space and the control space between the tube and the detector assembly, extracting feature points and feature point coordinates of the tube and the detector assembly, establishing a three-dimensional coordinate mapping of the feature points in the visual system coordinate system, calculating the partial derivatives of the image features with respect to the motion of the multi-degree-of-freedom manipulator through the three-dimensional coordinate mapping, and generating an image Jacobian matrix;

[0118] Collecting the real-time position coordinates and attitude angle of the detector assembly relative to the tube, calculating the position deviation vector and attitude deviation vector of the detector assembly relative to the tube, combining the position deviation vector and the attitude deviation vector to construct an error vector, determining a dynamic compensation control law based on the error vector and the image Jacobian matrix, and generating an initial control output of the multi-degree-of-freedom manipulator;

[0119] Based on the comparison result of the modulus value of the error vector and the preset error threshold, the adjustment direction of the proportional gain coefficient is determined, and according to the preset motion stability index of the multi-degree-of-freedom manipulator, the adjustment direction of the damping coefficient is determined, when the modulus value of the error vector is greater than the first error threshold, the proportional gain coefficient is reduced according to the preset proportion, and when the modulus value of the error vector is less than the second error threshold, the proportional gain coefficient is increased according to the preset proportion, and the control parameters after adaptive adjustment are generated;

[0120] In a preset observation time window, the relative feature change between the tube and the detector assembly and the joint motion increment of the multi-degree-of-freedom manipulator are collected, the relative feature change and the joint motion increment are input into a recursive least squares estimator to calculate a covariance matrix, a Kalman gain matrix is ​​calculated based on the covariance matrix, and the image Jacobian matrix is ​​updated based on the Kalman gain matrix to obtain a real-time image Jacobian matrix;

[0121] The initial control output and the real-time image Jacobian matrix are input into an auto-disturbance rejection observer, the external comprehensive interference term in the motion process of the multi-degree-of-freedom manipulator is calculated, and the compensation value of the external comprehensive interference term is superimposed on the initial control output to generate a compensation control output; the deviation distance between the actual tracking trajectory of the detector assembly relative to the tube and the expected tracking trajectory is calculated, an adaptive sliding mode control term is constructed according to the deviation distance, and the adaptive sliding mode control term is superimposed on the compensation control output to generate a sliding mode correction control output;

[0122] The operation is repeatedly performed in a plurality of sampling cycles to generate a final control instruction for adjusting the multi-degree-of-freedom manipulator.

[0123] In a specific implementation, first, a mapping relationship between the visual feature space and the control space between the tube and the detector assembly is established. Specifically, the feature points of the tube and the detector assembly, such as edges, corners, etc., are extracted, and the pixel coordinates of these feature points in the image are obtained. Then, using the camera calibration parameters, these pixel coordinates are converted into coordinates in the three-dimensional coordinate system of the visual system. In this way, a mapping relationship between the image features and the robotic arm motion space is established. For example, the image coordinates of the feature point A on the tube are (u1, v1), and its corresponding three-dimensional coordinates are (x1, y1, z1); the image coordinates of the feature point B on the detector assembly are (u2, v2), and its corresponding three-dimensional coordinates are (x2, y2, z2).

[0124] Next, the partial derivatives of the image features with respect to the robot arm motion are calculated to generate the image Jacobian matrix. The image Jacobian matrix describes the relationship between the change of the image features and the motion of the robot arm joints. For example, if the robot arm joint 1 moves a unit angle, how much will the coordinates of the image feature point A change? These partial derivatives can be calculated by numerical differentiation.

[0125] Then, the real-time position coordinates and attitude angles of the detector assembly relative to the tube are collected. These data can be obtained by sensors installed on the detector assembly. For example, the position coordinates can be expressed as (x, y, z) and the attitude angles can be expressed as (roll, pitch, yaw). The deviation of the expected position and attitude of the detector assembly relative to the tube is calculated to construct an error vector. For example, if the detector assembly is expected to be 10 cm above the tube and the actual distance is 12 cm, the position deviation is 2 cm.

[0126] Based on the error vector and the image Jacobian matrix, the dynamic compensation control law is determined to generate the initial control output of the robot. The control law can adopt methods such as proportional-differential control. For example, the initial control output can be the movement speed or angle change of each joint.

[0127] According to the modulus of the error vector and the preset error threshold, the proportional gain coefficient is adaptively adjusted. When the error is large, the proportional gain coefficient is reduced to ensure the stability of the system; when the error is small, the proportional gain coefficient is increased to improve the tracking accuracy. For example, the first error threshold is set to 5 cm and the second error threshold is set to 1 cm. When the error is greater than 5 cm, the proportional gain coefficient is reduced by half; when the error is less than 1 cm, the proportional gain coefficient is doubled. At the same time, the damping coefficient is adjusted according to the motion stability index of the robot arm.

[0128] In the preset observation time window, the relative characteristic changes between the tube and the detector assembly and the joint motion increment of the robot arm are collected. Using these data, the covariance matrix is ​​calculated by the recursive least squares estimator. Then, the Kalman gain matrix is ​​calculated based on the covariance matrix, and the image Jacobian matrix is ​​updated with the Kalman gain matrix to obtain the real-time image Jacobian matrix.

[0129] The initial control output and the real-time image Jacobian matrix are input into the ADAO to calculate the external comprehensive disturbance term during the motion of the robot. The compensation value of the external comprehensive disturbance term is superimposed on the initial control output to generate the compensated control output.

[0130] The deviation distance between the actual tracking trajectory of the detector assembly relative to the tube and the expected tracking trajectory is calculated. An adaptive sliding mode control term is constructed according to the deviation distance, and the adaptive sliding mode control term is superimposed on the compensation control output to generate a sliding mode correction control output.

[0131] The above operations are repeatedly performed within multiple sampling cycles to generate a final control instruction for adjusting the robotic arm, thereby controlling the movement of the robotic arm and allowing the detector assembly to accurately track the movement of the tube.

[0132] In this embodiment, through visual servo control and adaptive parameter adjustment, accurate tracking of the tube motion can be achieved and tracking errors can be reduced; the use of an anti-disturbance observer and sliding mode control can effectively suppress the impact of external interference and model uncertainty on tracking performance and improve the robustness of the system; through real-time updating of the image Jacobian matrix and control parameters, real-time tracking of the tube motion can be achieved to meet actual application needs.

[0133] In an optional implementation, inputting the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screening and obtaining the optimal image sequence data according to the image quality score data includes:

[0134] Input the multi-angle soft tissue projection data into the feature separation network, extract the detail features and structural features of the image respectively through the dual-branch structure, construct the frequency domain attention map respectively, enhance the detail features and structural features according to the frequency domain attention map, and obtain enhanced dual frequency domain feature data;

[0135] The dual-frequency domain feature data is input into a residual dense connection network including a plurality of residual dense blocks, a plurality of convolutional layers are connected in series in a dense connection manner in each residual dense block, and features are reused between adjacent residual dense blocks through skip connections to extract multi-level image features and obtain a multi-scale feature map;

[0136] Constructing a self-attention pyramid module, dividing the multi-scale feature map into feature blocks of different scales, calculating the correlation matrix between different feature blocks, generating attention weights according to the correlation matrix, adaptively weighting the feature blocks, and obtaining fused feature data;

[0137] Inputting the fused feature data into an uncertainty learning module, wherein the uncertainty learning module comprises two parallel branches, outputting a quality score prediction value through a first branch, outputting a prediction uncertainty estimate value through a second branch, combining the quality score prediction value and the prediction uncertainty estimate value to generate a credibility-weighted quality score;

[0138] Sort the images according to the credibility-weighted quality scores to obtain an initial candidate sequence, perform pairwise similarity calculation on the images in the initial candidate sequence, and construct a similarity matrix; perform cluster analysis on the initial candidate sequence based on the similarity matrix, classify images with similarities higher than a preset similarity threshold into the same category, and select the image with the highest quality score in each category as a category labeled image;

[0139] The mutual information between each image in the initial candidate sequence and the corresponding category labeled image is calculated, and the images are reordered according to a weighted combination of the mutual information and the quality score. According to a preset front row ratio, images with corresponding rankings and meeting the minimum image quantity requirement are selected to form an optimal image sequence.

[0140] In a specific embodiment, the image quality assessment network is used to perform a quality score on each multi-angle soft tissue projection image input. First, the image is input into a feature separation network. The network adopts a dual-branch structure to extract detail features and structural features of the image respectively. Detail features focus on the high-frequency information of the image, such as edges, textures, etc.; structural features focus on the low-frequency information of the image, such as contours, shapes, etc. A frequency domain attention map is constructed for detail features and structural features respectively. The frequency domain attention map is used to enhance important frequency components and suppress irrelevant information such as noise. For example, for detail features, the high-frequency part can be enhanced to highlight edge information; for structural features, the low-frequency part can be enhanced to highlight shape information. The frequency domain attention map is applied to the corresponding features to obtain enhanced dual-frequency domain feature data.

[0141] The enhanced dual-frequency domain feature data is input into the residual dense connection network. The network contains multiple residual dense blocks. Each residual dense block uses dense connection to connect the outputs of all convolutional layers in the block in series, making full use of feature information at different levels. Adjacent residual dense blocks reuse features through jump connections to further improve the network's expression ability. For example, if the first residual dense block contains three convolutional layers, the outputs of these three convolutional layers are connected in series. In addition to its own input, the input of the second residual dense block also includes the output of the first residual dense block. In this way, multi-level image features are extracted to obtain multi-scale feature maps.

[0142] The multi-scale feature map is input into the self-attention pyramid module. This module divides the feature map into feature blocks of different scales. For example, the feature map can be divided into blocks of 1×1, 2×2, 4×4, etc. The correlation matrix between different feature blocks is calculated to reflect the semantic relationship between different blocks. The attention weights are generated according to the correlation matrix, and the feature blocks are adaptively weighted. For example, if the correlation between two blocks is high, a larger weight is given, and vice versa. The weighted feature blocks are fused to obtain fused feature data.

[0143] The fused feature data is fed into the uncertainty learning module. This module contains two parallel branches. The first branch outputs a quality score prediction, such as an image quality score of 0.8. The second branch outputs a prediction uncertainty estimate, such as an uncertainty of 0.1. The quality score prediction and prediction uncertainty estimate are combined to generate a credibility-weighted quality score. For example, the credibility-weighted quality score can be calculated as 0.8×(1-0.1)=0.72.

[0144] Sort the images according to the credibility-weighted quality score to obtain the initial candidate sequence. Assume there are 10 images, and sort them from high to low. Perform pairwise similarity calculations on the images in the initial candidate sequence to construct a similarity matrix. For example, the similarity between the first image and the second image is calculated to be 0.9, the similarity between the first image and the third image is 0.7, and so on, to construct a 10×10 similarity matrix. Perform cluster analysis on the initial candidate sequence based on the similarity matrix. Classify images with similarity higher than the preset similarity threshold into the same category. For example, if the threshold is set to 0.8, images with similarity higher than 0.8 are classified into the same category. Select the image with the highest quality score in each category as the category marker image.

[0145] Calculate the mutual information between each image in the initial candidate sequence and the corresponding category labeled image. For example, calculate the mutual information between the first image and the corresponding category labeled image. Reorder the images according to the weighted combination of mutual information and quality score. For example, the mutual information and quality score can be multiplied by 0.4 and 0.6 respectively and then added to obtain a new sorting basis. According to the preset front row ratio, such as the top 30%, select the images with corresponding ranking and meeting the minimum number of images required to form the optimal image sequence. Assuming that the minimum number of images required is 3, select the top 3 images to form the optimal image sequence.

[0146] In this embodiment, through techniques such as feature separation, frequency domain attention mechanism and uncertainty learning, image quality can be evaluated more accurately and the impact of noise and artifacts can be reduced; through similarity clustering and mutual information calculation, the most representative and informative image subset can be selected to reduce redundant data and improve the efficiency of three-dimensional reconstruction; by selecting high-quality image sequences, the accuracy of three-dimensional reconstruction can be effectively improved to obtain a clearer and more accurate three-dimensional model.

[0147] In an optional implementation, matching the soft tissue structure feature data with the inertial sensor data at feature points, and generating the registered image data by a self-correcting spatial registration algorithm comprises:

[0148] In the soft tissue structural feature data, a circular neighborhood with a preset radius is constructed for each candidate feature point, and the grayscale difference between the pixel point in the circular neighborhood and the center point and the grayscale variance of the circular neighborhood are calculated. When the grayscale difference of a plurality of consecutive pixel points is greater than a preset grayscale difference threshold, and the grayscale variance is greater than a preset grayscale variance threshold, the candidate feature point is marked as a structural feature point, and a soft tissue feature point set is obtained;

[0149] Performing Kalman filter preprocessing on the inertial sensor data to obtain filtered data, performing spatial integration on the filtered data to obtain sensor position coordinates, and projecting them onto an image plane to obtain a position feature point set; calculating the Mahalanobis distance for each position feature point in the position feature point set, eliminating position feature points whose Mahalanobis distance is greater than a preset distance threshold, and obtaining an optimized position feature point set;

[0150] Calculating the matrix eigenvalue ratio of the neighborhood corresponding to each feature point in the soft tissue feature point set and the optimized position feature point set to obtain the local structure complexity;

[0151] Adaptively adjusting the sampling radius according to the local structure complexity, constructing a multi-scale sampling pattern around each feature point, and extracting feature description vectors of each feature point in the soft tissue feature point set and the optimized position feature point set based on the multi-scale sampling pattern;

[0152] Calculating the mutual information coefficient and the relative position relationship parameter between the feature description vectors, matching the soft tissue feature point set and the optimized position feature point set according to the mutual information coefficient and the relative position relationship parameter, and obtaining an initial matching point pair set;

[0153] Calculate the mutual information coefficient of the feature description vector and the consistency degree of local space transformation of each pair of matching points in the initial matching point pair set, and generate a matching credibility score in combination with the motion constraints provided by the inertial sensor data; screen the initial matching point pair set according to the matching credibility score, and select the matching point pairs whose matching credibility scores are higher than a preset matching credibility score threshold as the preferred matching point pairs; construct a spatial transformation matrix based on the preferred matching point pairs, and adaptively determine the loss function threshold parameter according to the current matching error distribution;

[0154] Based on the loss function threshold parameter, combined with the Huber loss function, the parameters of the spatial transformation matrix are optimized, and when the parameter change of the spatial transformation matrix during the optimization process is less than a preset matrix parameter change threshold, the current spatial transformation matrix is ​​determined as the optimal transformation matrix;

[0155] The soft tissue structure feature data is spatially transformed according to the optimal transformation matrix to generate registration image data.

[0156] The Huber loss function specifically refers to a loss function for regression problems, which combines the advantages of mean square error loss and absolute error loss. Specifically, when the error is small, it is expressed as mean square error, and when the error is large, it is expressed as absolute error, so that it can be sensitive to small errors and robust to large errors (i.e., reduce the impact of outliers). It can be defined in sections as follows: when the error is small (within the threshold), the loss is the square term of the error, similar to the mean square error, which makes it more sensitive to small errors and can converge faster. The loss function is in quadratic form and appears as a smooth curve; when the error is large (exceeding the threshold), the loss is a linear term of the absolute error, which reduces the impact of outliers, and the growth rate of the loss function increases linearly, rather than increasing rapidly like the mean square error.

[0157] In a specific embodiment, first, extract soft tissue structural feature points. For the input soft tissue structural feature data (for example, ultrasound images), traverse each pixel point and use it as a candidate feature point. With each candidate feature point as the center, construct a circular neighborhood with a preset radius (for example, 5 pixels). Calculate the grayscale difference between all pixels in the circular neighborhood and the center point, as well as the grayscale variance of the circular neighborhood. Set the grayscale difference threshold and the grayscale variance threshold (for example, the grayscale difference threshold is 10, and the grayscale variance threshold is 20). If there are multiple consecutive pixels (for example, 3 pixels) in the circular neighborhood whose grayscale difference is greater than the preset grayscale difference threshold, and the grayscale variance of the circular neighborhood is also greater than the preset grayscale variance threshold, then mark the candidate feature point as a structural feature point. After traversing all candidate feature points, a set of soft tissue feature points is obtained. For example, in a grayscale image, in the circular neighborhood of the pixel point (10, 10), the grayscale differences of three consecutive pixels are 12, 15, and 13, respectively, all greater than 10, and the grayscale variance of the neighborhood is 25, which is greater than 20, so (10, 10) is marked as a structural feature point.

[0158] Next, process the inertial sensor data. Perform Kalman filter preprocessing on the acquired inertial sensor data (e.g., accelerometer and gyroscope data) to remove noise and drift. After obtaining the filtered data, perform spatial integration on it to calculate the position coordinates of the sensor at each time point. Project these position coordinates onto the image plane corresponding to the soft tissue structure feature data to obtain a set of position feature points. In order to eliminate abnormal data, calculate the Mahalanobis distance for each position feature point in the position feature point set. Set a preset distance threshold (e.g., 3). If the Mahalanobis distance of a position feature point is greater than the threshold, it is eliminated. Finally, the optimized position feature point set is obtained. For example, the Mahalanobis distance of the position feature point (5, 5) is 4, which is greater than the threshold 3, so it is eliminated.

[0159] Then, the local structural complexity is calculated. For each feature point in the soft tissue feature point set and the optimized position feature point set, the matrix eigenvalue ratio of its corresponding neighborhood is calculated as a measure of the local structural complexity. Specifically, a neighborhood (for example, a 5×5 window) is constructed with each feature point as the center. The covariance matrix of the pixel values ​​in the neighborhood is calculated, and the eigenvalues ​​of the covariance matrix are calculated. The ratio of the maximum eigenvalue to the minimum eigenvalue is taken as the local structural complexity of the feature point.

[0160] Adaptively adjust the sampling radius according to the local structural complexity. For each feature point, the sampling radius is adaptively adjusted according to its local structural complexity. The higher the local structural complexity, the larger the sampling radius. Construct a multi-scale sampling pattern around each feature point. For example, if the local structural complexity of the feature point is 10, the sampling radius is 3; if the local structural complexity is 20, the sampling radius is 5. Based on the multi-scale sampling pattern, extract the feature description vectors of each feature point in the soft tissue feature point set and the optimized position feature point set. For example, the feature description vector can be extracted using a local binary pattern (LBP).

[0161] Next, feature matching is performed. The mutual information coefficient and relative position relationship parameter between each feature description vector in the soft tissue feature point set and the optimized position feature point set are calculated. The mutual information coefficient is used to measure the similarity between two feature vectors, and the relative position relationship parameter is used to describe the spatial relationship between two feature points. According to the mutual information coefficient and the relative position relationship parameter, the soft tissue feature point set and the optimized position feature point set are matched to obtain the initial matching point pair set.

[0162] Screen the initial matching point pairs. Calculate the mutual information coefficient of the feature description vector and the consistency of the local space transformation for each pair of matching points in the initial matching point pair set. Combined with the motion constraints provided by the inertial sensor data, generate a matching credibility score. For example, if a pair of matching points has a high mutual information coefficient, a high consistency of the local space transformation, and meets the motion constraints provided by the inertial sensor data, then its matching credibility score is high. Screen the initial matching point pair set according to the matching credibility score, and select the matching point pairs with a matching credibility score higher than a preset threshold (for example, 0.8) as the preferred matching point pairs.

[0163] Perform spatial transformation and optimization. Construct a spatial transformation matrix based on the preferred matching point pairs. Adaptively determine the loss function threshold parameter based on the current matching error distribution. For example, if the matching error is large, the loss function threshold parameter is large. Combined with the Huber loss function, perform parameter optimization on the spatial transformation matrix. When the parameter change of the spatial transformation matrix during the optimization process is less than the preset matrix parameter change threshold (for example, 0.001), the current spatial transformation matrix is ​​determined as the optimal transformation matrix.

[0164] Finally, the registration image data is generated. The soft tissue structure feature data is spatially transformed according to the optimal transformation matrix to generate the registration image data, thus realizing the registration of the soft tissue structure feature data and the inertial sensor data.

[0165] In this embodiment, by combining the local structure complexity and inertial sensor data, soft tissue feature points and position feature points can be matched more accurately, thereby improving the registration accuracy; the use of adaptive sampling radius and multi-scale sampling mode can better adapt to soft tissue structures of different complexities and enhance the robustness of the registration; by adaptively determining the loss function threshold parameters, the optimization speed of the spatial transformation matrix can be accelerated and the computational efficiency can be improved.

[0166] In an optional implementation, the optimized multi-angle image data and the material characteristic data are input into a compressed sensing reconstruction model, regularized constraints are constructed in combination with a human soft tissue anatomical structure template, and soft tissue density distribution data is generated through iterative calculation, including:

[0167] Obtaining photon attenuation coefficients according to material characteristic data, constructing a tissue attenuation coefficient matrix, establishing a linear density mapping function based on the tissue attenuation coefficient matrix, and obtaining a material-density mapping model;

[0168] Constructing an orthogonal sparse basis matrix based on the optimized multi-angle image data, and calculating the sparse representation coefficients of the projection data under the orthogonal sparse basis matrix;

[0169] Extracting tissue interface position coordinates and interface curvature values ​​from a human soft tissue anatomical structure template, and constructing anatomical feature constraints based on the interface position coordinates and the interface curvature values;

[0170] Using the interface position coordinates, the density reconstruction space is divided into a plurality of continuous sub-regions, a density continuity constraint is imposed inside the continuous sub-regions, and a piecewise continuous constraint function with controllable jumps is constructed at the interface position coordinates to generate a density distribution regularization term;

[0171] Calculating structural similarity coefficients of adjacent image blocks in the reconstruction space, constructing a non-local difference operator according to the structural similarity coefficient, applying the non-local difference operator to density distribution data, and establishing a structure preservation constraint term;

[0172] The material-density mapping model, the anatomical feature constraint, the density distribution regularization term and the structure preservation constraint term are combined into a joint optimization objective function; the joint optimization objective function is decomposed into variables by an augmented Lagrangian method to obtain a projection consistency constraint term and a structure preservation constraint term;

[0173] Based on the projection consistency constraint, the projection coefficient matrix is ​​solved by the conjugate gradient method, and based on the structure preservation constraint, the constraint parameters are optimized by the proximal gradient descent method to obtain the density distribution image of the current iteration;

[0174] A three-layer dictionary learning network is constructed based on historical reconstruction data, and feature dictionary groups with different scale organizational structures are obtained through training.

[0175] Sparsely decompose the density distribution image of the current iteration in the feature dictionary group, calculate the reconstruction error at each scale, select the feature dictionary with the smallest reconstruction error to reconstruct the density distribution, and obtain an optimized density image;

[0176] The Euclidean distance difference between the optimized density images obtained in two adjacent iterations is calculated, and when the Euclidean distance difference is less than a preset density image distance difference threshold, the current optimized density image is determined as the soft tissue density distribution data.

[0177] The photon attenuation coefficient specifically refers to a physical quantity that describes the rate at which the number of photons decreases due to various interactions (such as absorption, scattering, etc.) when light passes through a substance. It reflects the ability of a substance to attenuate photons and is one of the inherent characteristics of a substance. The photon attenuation coefficient indicates the degree of attenuation of photon intensity per unit length. It is the sum of the probabilities of photons being absorbed or scattered when a substance is irradiated by photons of a certain energy. It is generally measured in units of per centimeter. The size of the photon attenuation coefficient depends on the energy of the photon and the chemical composition and density of the substance. A high attenuation coefficient indicates that a substance absorbs or scatters photons strongly, and the ability of photons to penetrate it is weak.

[0178] In a specific implementation, first, characteristic data of the material is obtained, such as attenuation coefficients of different tissue types. These attenuation coefficients are constructed into a matrix, called a tissue attenuation coefficient matrix. Through this matrix, a linear mapping function is established to map the material type to the corresponding density value, thereby obtaining a material-density mapping model.

[0179] Then, the collected multi-angle image data is optimized, such as removing noise and artifacts. Using these optimized multi-angle image data, an orthogonal sparse basis matrix is ​​constructed. The projection data is decomposed under this orthogonal sparse basis matrix to obtain sparse representation coefficients.

[0180] Next, the tissue interface position coordinates and interface curvature values ​​are extracted from the human soft tissue anatomical structure template. For example, the boundary information of different tissues such as bones, muscles, and fat can be extracted from the template using a segmentation algorithm. Based on the extracted interface position coordinates and curvature values, anatomical feature constraints are constructed to constrain the shape and structure of the reconstruction results.

[0181] Using the extracted interface position coordinates, the density reconstruction space is divided into multiple continuous sub-regions. Within each sub-region, a density continuity constraint is imposed, such as requiring that the density value difference between adjacent voxels cannot be too large. At the tissue interface position, a piecewise continuous constraint function is constructed to allow the density value to jump at the interface, but the jump amplitude must be controlled. These constraints are combined to generate a density distribution regularization term.

[0182] Calculate the structural similarity coefficients of adjacent image blocks in the reconstruction space. For example, the similarity of the gradient direction histograms of adjacent image blocks can be calculated. According to the calculated structural similarity coefficients, a non-local difference operator is constructed. The non-local difference operator is applied to the density distribution data to establish a structure preservation constraint term to preserve the texture details of the reconstruction result.

[0183] The material-density mapping model, anatomical feature constraints, density distribution regularization terms and structure preservation constraints obtained above are combined into a joint optimization objective function. The augmented Lagrangian method is used to decompose the variables of this joint optimization objective function to obtain the projection consistency constraint term and the structure preservation constraint term.

[0184] Based on the projection consistency constraint, the conjugate gradient method is used to solve the projection coefficient matrix. Based on the structure preservation constraint, the proximal gradient descent method is used to optimize the constraint parameters. By iteratively updating the projection coefficient matrix and constraint parameters, the density distribution image of the current iteration is obtained.

[0185] In order to further improve the reconstruction quality, a three-layer dictionary learning network is constructed. The network is trained using historical reconstruction data to learn feature dictionary groups of organizational structures at different scales. The density distribution image of the current iteration is sparsely decomposed in these feature dictionary groups, and the reconstruction error at each scale is calculated. The feature dictionary with the smallest reconstruction error is selected to reconstruct the density distribution and obtain the optimized density image.

[0186] The Euclidean distance difference between the optimized density images obtained in two adjacent iterations is calculated. When the Euclidean distance difference is less than a preset density image distance difference threshold, the current optimized density image is determined as the final soft tissue density distribution data. For example, the preset threshold can be set to 0.001.

[0187] Assume that the material characteristic data includes three tissue types: bone, muscle and fat, and their corresponding attenuation coefficients are 0.2, 0.1, and 0.05, respectively. The multi-angle image data includes projection data collected from three angles: 0 degree, 45 degree and 90 degree. The human soft tissue anatomical structure template includes segmented images of bone, muscle and fat. Through the above method, a three-dimensional density distribution image containing three tissues of bone, muscle and fat can be reconstructed.

[0188] In this embodiment, by combining multi-angle image data, material feature data and anatomical structure information, the accuracy of soft tissue density distribution reconstruction is effectively improved; by introducing structure-preserving constraints, the tissue structure and texture details of the reconstruction results can be effectively maintained; by adopting the augmented Lagrangian method and dictionary learning technology, the reconstruction process can be effectively accelerated.

[0189] In an optional implementation, constructing a tissue classification model based on the topological feature data and the morphological feature data, intelligently segmenting different density regions, and generating three-dimensional model data containing tissue layering information includes:

[0190] Inputting the topological feature data and the morphological feature data into a three-dimensional convolution extraction network to obtain a multi-scale feature mapping sequence, constructing a feature hierarchy network based on the multi-scale feature mapping sequence, calculating a local structure tensor matrix, extracting a tissue boundary direction vector and a principal curvature parameter from the local structure tensor matrix, and generating a structure description vector;

[0191] Using the structure description vector to construct an organizational connectivity graph model, wherein the graph nodes of the organizational connectivity graph model store the local structure description vector, and the graph edges store the spatial position relationship between adjacent nodes;

[0192] Constructing a feature enhancement unit based on phase change diffusion, inputting the tissue connectivity graph model into a nonlinear diffusion equation, solving the nonlinear diffusion equation, obtaining enhanced boundary features and enhanced internal structure features, and updating node features of the tissue connectivity graph model;

[0193] Establish a soft tissue elastic mechanics model based on domain knowledge, convert the tissue stress-strain relationship into node displacement constraint parameters, generate biomechanical constraint conditions, calculate tissue deformation strain energy according to the biomechanical constraint conditions, and construct a regional growth function based on the tissue deformation strain energy minimization criterion;

[0194] The tissue connectivity graph model is segmented iteratively and hierarchically according to the regional growth function to obtain segmented regions, the values ​​of tissue deformation strain energy of the segmented regions are calculated, and the boundaries where the difference in tissue deformation strain energy of adjacent regions is greater than a preset strain energy difference threshold are determined as tissue demarcation point sets; a tissue hierarchy label set is constructed based on the tissue demarcation point set, a tissue hierarchy rule base is established, and tissue hierarchy data is generated;

[0195] Performing credibility calculation on the organizational hierarchy data to obtain a regional classification credibility value, and marking the regions whose regional classification credibility values ​​are lower than a preset credibility threshold as a set of regions to be optimized;

[0196] Adopting a neighborhood information transfer algorithm to iteratively optimize the set of regions to be optimized, updating node feature parameters and edge constraint parameters, until the classification credibility values ​​of all regions are greater than the preset credibility threshold, and obtaining optimized classification identification data;

[0197] The optimized classification identification data and the density distribution data are spatially registered and fused to generate a three-dimensional soft tissue model with anatomical structure annotations.

[0198] In a specific embodiment, first, the acquired tissue topological feature data (such as tissue surface mesh data, topological connection relationship) and morphological feature data (such as tissue density, fiber orientation, grayscale value) are input into a three-dimensional convolution extraction network. The network can be a pre-trained model, such as 3D U-Net or V-Net, or a customized network designed for a specific tissue type. Taking a tissue block containing 512×512×256 voxels as an example, after a three-dimensional convolution extraction network, a series of feature mapping sequences of different scales can be obtained, such as 128×128×64, 64×64×32, 32×32×16, etc.

[0199] Next, a feature hierarchy network is constructed based on these multi-scale feature map sequences. The network can be a pyramid structure that fuses feature maps of different scales to capture the global and local features of the tissue. At each layer of the feature hierarchy network, a local structure tensor matrix is ​​calculated. This matrix can reflect the shape and direction information of the local area of ​​the tissue. For example, the local structure tensor matrix can be constructed by calculating the gradient information within the neighborhood of each voxel. The tissue boundary direction vector and the principal curvature parameters are extracted from the local structure tensor matrix. These parameters can describe the degree and direction of the curvature of the tissue surface. The extracted boundary direction vector and principal curvature parameters are combined into a structure description vector. For example, a structure description vector can contain three components of the boundary direction and two principal curvature values.

[0200] Then, these structural description vectors are used to construct a tissue connectivity graph model. The graph nodes of this model store local structural description vectors, and the graph edges store the spatial position relationship between adjacent nodes. For example, each voxel can be regarded as a graph node, and graph edges are established between adjacent voxels. Taking an 8×8×8 local area as an example, a connectivity graph model containing 512 nodes can be constructed.

[0201] Construct a feature enhancement unit based on phase-change diffusion. Input the tissue connectivity graph model into the nonlinear diffusion equation. By iteratively solving the equation, the tissue boundary features and internal structure features can be enhanced. For example, a nonlinear diffusion equation based on the Perona-Malik model can be used, which can adaptively adjust the diffusion coefficient according to the image gradient size, thereby retaining edge information while smoothing the area. After solving the nonlinear diffusion equation, enhanced boundary features and enhanced internal structure features are obtained, and these enhanced features are used to update the node features of the tissue connectivity graph model.

[0202] A soft tissue elastic mechanics model is established based on domain knowledge (e.g., elastic modulus and Poisson's ratio of tissue). The tissue stress-strain relationship is converted into node displacement constraint parameters to generate biomechanical constraints. For example, the elastic modulus of the tissue can be set to 10 kPa and the Poisson's ratio can be set to 0.45. Tissue deformation strain energy is calculated according to biomechanical constraints. For example, the finite element method can be used to calculate tissue deformation strain energy. A regional growth function based on the criterion of minimizing tissue deformation strain energy is constructed. This function is used to guide the subsequent tissue segmentation process.

[0203] The tissue connectivity graph model is segmented iteratively and hierarchically according to the regional growth function to obtain the segmented regions. The value of the tissue deformation strain energy of the segmented region is calculated. The boundary where the difference in tissue deformation strain energy between adjacent regions is greater than a preset strain energy difference threshold (e.g., 0.1 J) is determined as a tissue demarcation point set. A tissue hierarchy label set is constructed based on the tissue demarcation point set, a tissue hierarchy rule base is established, and tissue hierarchy data is generated. For example, different levels of tissue can be labeled as "epidermis", "dermis", "subcutaneous tissue", etc.

[0204] The credibility of the organizational hierarchy data is calculated to obtain the regional classification credibility value. For example, the credibility value can be calculated based on the consistency of the internal features of the region. The regions whose regional classification credibility values ​​are lower than the preset credibility threshold (for example, 0.8) are marked as the set of regions to be optimized.

[0205] The neighborhood information transfer algorithm is used to iteratively optimize the set of regions to be optimized, and the node feature parameters and edge constraint parameters are updated until the classification credibility values ​​of all regions are greater than the preset credibility threshold. For example, a neighborhood information transfer algorithm based on a graph convolutional network can be used to obtain optimized classification identification data.

[0206] Finally, the optimized classification identification data and density distribution data are spatially registered and fused to generate a 3D soft tissue model with anatomical structure annotations. For example, the classification identification of different tissues can be mapped to different colors to intuitively display the tissue structure in the 3D model.

[0207] In this embodiment, by combining topological and morphological features and introducing biomechanical constraints, tissue boundaries can be identified more accurately, especially in areas with smaller density differences, thereby improving the accuracy of tissue segmentation; hierarchical segmentation can be performed according to the hierarchical structure of the tissue, and a three-dimensional model containing tissue hierarchy information can be generated, which helps to gain a deeper understanding of the structure and function of the tissue; through credibility calculation and iterative optimization, errors in the segmentation results can be identified and corrected, thereby improving the reliability and credibility of the model.

[0208] Figure 2 FIG. 1 is a schematic diagram of a multi-angle photography and three-dimensional reconstruction system for soft tissue according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0209] The first unit is used to perform feature analysis on the inspected soft tissue using a deep convolutional neural network to obtain volume parameters and density distribution parameters of the soft tissue; based on the volume parameters and density distribution parameters, generate tube motion trajectory data and sampling interval data through a trajectory optimization algorithm; control a multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjust the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller; control the tube to collect soft tissue projection images in a preset first energy level range and a second energy level range respectively according to the sampling interval data, and obtain a multi-angle soft tissue projection data set;

[0210] The second unit is used to input the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screen and obtain optimal image sequence data according to the image quality score data; perform material decomposition processing on the optimal image sequence data, and use a multi-scale feature fusion network to extract soft tissue structure feature data and material feature data; perform feature point matching on the soft tissue structure feature data and inertial sensor data, and generate registration image data through a self-correction spatial registration algorithm; and perform adaptive noise reduction processing and local contrast enhancement processing based on wavelet transform on the registration image data in sequence to obtain optimized multi-angle image data;

[0211] The third unit is used to input the optimized multi-angle image data and the material feature data into a compressed sensing reconstruction model, build regularized constraints in combination with a human soft tissue anatomical structure template, and generate soft tissue density distribution data through iterative calculation; based on the soft tissue density distribution data, an adaptive meshing algorithm is used to perform surface reconstruction to generate initial three-dimensional model data; the initial three-dimensional model data is input into a graph convolutional feature extraction network to extract topological feature data and morphological feature data; a tissue classification model is built based on the topological feature data and the morphological feature data, different density areas are intelligently segmented, and three-dimensional model data containing tissue stratification information is generated.

[0212] According to a third aspect of the embodiments of the present invention,

[0213] An electronic device is provided, comprising:

[0214] processor;

[0215] a memory for storing processor-executable instructions;

[0216] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0217] According to a fourth aspect of the embodiments of the present invention,

[0218] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0219] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-angle photography and three-dimensional reconstruction method for soft tissue, characterized in that: include: A deep convolutional neural network is used to perform feature analysis on the soft tissue under examination to obtain volume parameters and density distribution parameters of the soft tissue; based on the volume parameters and density distribution parameters, a trajectory optimization algorithm is used to generate tube motion trajectory data and sampling interval data; a multi-degree-of-freedom manipulator is controlled to drive the tube to move according to the tube motion trajectory data, and a visual servo controller is used to adjust the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking; according to the sampling interval data, the tube is controlled to collect soft tissue projection images in a preset first energy level range and a second energy level range, respectively, to obtain a multi-angle soft tissue projection data set; Inputting the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screening to obtain optimal image sequence data according to the image quality score data; The optimal image sequence data is subjected to material decomposition processing, and soft tissue structure feature data and material feature data are extracted using a multi-scale feature fusion network; the soft tissue structure feature data is matched with inertial sensor data for feature points, and registered image data is generated using a self-correcting spatial registration algorithm; the registered image data is subjected to adaptive noise reduction processing and local contrast enhancement processing based on wavelet transform in sequence to obtain optimized multi-angle image data; Input the optimized multi-angle image data and the material feature data into a compressed sensing reconstruction model, build regularized constraint conditions in combination with a human soft tissue anatomical structure template, and generate soft tissue density distribution data through iterative calculation; based on the soft tissue density distribution data, use an adaptive meshing algorithm to perform surface reconstruction to generate initial three-dimensional model data; input the initial three-dimensional model data into a graph convolution feature extraction network to extract topological feature data and morphological feature data; build a tissue classification model based on the topological feature data and the morphological feature data, perform intelligent segmentation on different density areas, and generate three-dimensional model data containing tissue stratification information; Based on volume parameters and density distribution parameters, the trajectory optimization algorithm generates tube motion trajectory data and sampling interval data including: Determine a spatial envelope region based on the volume parameter, establish a three-dimensional grid in the spatial envelope region, recursively refine the three-dimensional grid using an adaptive grid division strategy, determine the degree of grid refinement by calculating a density curvature value and a gradient change rate of the grid unit, subdivide the grid unit when the density curvature value is greater than a first preset threshold or the gradient change rate is greater than a second preset threshold, and merge the grid units when both the density curvature value and the gradient change rate are less than a third preset threshold; Constructing a density feature tensor, wherein the density feature tensor includes a density value, a density gradient value, and a density Hessian matrix eigenvalue of a grid node, calculating an importance score of the grid node based on the density feature tensor, selecting grid nodes whose importance scores are greater than a preset threshold as candidate sampling nodes, performing hierarchical clustering on the candidate sampling nodes, and introducing a topological constraint in each cluster to determine a key sampling point; Establish an imaging quality evaluation model that combines ray attenuation and scattering effects, add the image clarity index and signal-to-noise ratio index output by the imaging quality evaluation model to the path optimization objective function, construct an optimization problem of the optimal access sequence based on the key sampling points, and use an iterative optimization algorithm with an adaptive step size to solve the optimal path connecting the key sampling points; A local search strategy is introduced in the optimization process. When a local suboptimal solution is detected in the connection path between the key sampling points, the local path segment is reconstructed based on the taboo search method, and the reconstructed path is corrected using the path smoothness constraint to generate the ball tube motion trajectory data connecting all the key sampling points. The path between adjacent key sampling points is divided into a plurality of sampling areas according to the tube motion trajectory data, a density gradient histogram is established in each sampling area, an initial value of the sampling interval is determined based on the statistical characteristics of the density gradient histogram, and the sampling interval is adaptively adjusted in combination with the degree of tissue interface transition of the key sampling points to generate the sampling interval data.

2. The method according to claim 1, characterized in that Controlling the multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjusting the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller includes: Constructing a mapping relationship between the visual feature space and the control space between the tube and the detector assembly, extracting feature points and feature point coordinates of the tube and the detector assembly, establishing a three-dimensional coordinate mapping of the feature points in the visual system coordinate system, calculating the partial derivatives of the image features with respect to the motion of the multi-degree-of-freedom manipulator through the three-dimensional coordinate mapping, and generating an image Jacobian matrix; Collecting the real-time position coordinates and attitude angle of the detector assembly relative to the tube, calculating the position deviation vector and attitude deviation vector of the detector assembly relative to the tube, combining the position deviation vector and the attitude deviation vector to construct an error vector, determining a dynamic compensation control law based on the error vector and the image Jacobian matrix, and generating an initial control output of the multi-degree-of-freedom manipulator; Based on the comparison result of the modulus value of the error vector and the preset error threshold, the adjustment direction of the proportional gain coefficient is determined, and according to the preset motion stability index of the multi-degree-of-freedom manipulator, the adjustment direction of the damping coefficient is determined, when the modulus value of the error vector is greater than the first error threshold, the proportional gain coefficient is reduced according to the preset proportion, and when the modulus value of the error vector is less than the second error threshold, the proportional gain coefficient is increased according to the preset proportion, and the control parameters after adaptive adjustment are generated; In a preset observation time window, the relative feature change between the tube and the detector assembly and the joint motion increment of the multi-degree-of-freedom manipulator are collected, the relative feature change and the joint motion increment are input into a recursive least squares estimator to calculate a covariance matrix, a Kalman gain matrix is ​​calculated based on the covariance matrix, and the image Jacobian matrix is ​​updated based on the Kalman gain matrix to obtain a real-time image Jacobian matrix; The initial control output and the real-time image Jacobian matrix are input into an auto-disturbance rejection observer, the external comprehensive interference term in the motion process of the multi-degree-of-freedom manipulator is calculated, and the compensation value of the external comprehensive interference term is superimposed on the initial control output to generate a compensation control output; the deviation distance between the actual tracking trajectory of the detector assembly relative to the tube and the expected tracking trajectory is calculated, an adaptive sliding mode control term is constructed according to the deviation distance, and the adaptive sliding mode control term is superimposed on the compensation control output to generate a sliding mode correction control output; The operation is repeatedly performed in a plurality of sampling cycles to generate a final control instruction for adjusting the multi-degree-of-freedom manipulator.

3. The method according to claim 1, characterized in that Inputting the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screening and obtaining optimal image sequence data according to the image quality score data includes: Input the multi-angle soft tissue projection data into the feature separation network, extract the detail features and structural features of the image respectively through the dual-branch structure, construct the frequency domain attention map respectively, enhance the detail features and structural features according to the frequency domain attention map, and obtain enhanced dual frequency domain feature data; The dual-frequency domain feature data is input into a residual dense connection network including a plurality of residual dense blocks, a plurality of convolutional layers are connected in series in a dense connection manner in each residual dense block, and features are reused between adjacent residual dense blocks through skip connections, multi-level image features are extracted, and a multi-scale feature map is obtained; Constructing a self-attention pyramid module, dividing the multi-scale feature map into feature blocks of different scales, calculating the correlation matrix between different feature blocks, generating attention weights according to the correlation matrix, adaptively weighting the feature blocks, and obtaining fused feature data; Inputting the fused feature data into an uncertainty learning module, the uncertainty learning module comprising two parallel branches, outputting a quality score prediction value through a first branch, outputting a prediction uncertainty estimate value through a second branch, combining the quality score prediction value and the prediction uncertainty estimate value to generate a credibility-weighted quality score; Sort the images according to the credibility-weighted quality scores to obtain an initial candidate sequence, perform pairwise similarity calculation on the images in the initial candidate sequence, and construct a similarity matrix; perform cluster analysis on the initial candidate sequence based on the similarity matrix, classify images with similarities higher than a preset similarity threshold into the same category, and select the image with the highest quality score in each category as a category labeled image; The mutual information between each image in the initial candidate sequence and the corresponding category labeled image is calculated, and the images are reordered according to a weighted combination of the mutual information and the quality score. According to a preset front row ratio, images with corresponding rankings and meeting the minimum image quantity requirement are selected to form an optimal image sequence.

4. The method according to claim 1, characterized in that: Matching the soft tissue structure feature data with the inertial sensor data at feature points, and generating registration image data by a self-correction spatial registration algorithm includes: In the soft tissue structural feature data, a circular neighborhood with a preset radius is constructed for each candidate feature point, and the grayscale difference between the pixel point in the circular neighborhood and the center point and the grayscale variance of the circular neighborhood are calculated. When the grayscale difference of a plurality of consecutive pixel points is greater than a preset grayscale difference threshold, and the grayscale variance is greater than a preset grayscale variance threshold, the candidate feature point is marked as a structural feature point, and a soft tissue feature point set is obtained; Performing Kalman filter preprocessing on the inertial sensor data to obtain filtered data, performing spatial integration on the filtered data to obtain sensor position coordinates, and projecting them onto an image plane to obtain a position feature point set; calculating the Mahalanobis distance for each position feature point in the position feature point set, eliminating position feature points whose Mahalanobis distance is greater than a preset distance threshold, and obtaining an optimized position feature point set; Calculating the matrix eigenvalue ratio of the neighborhood corresponding to each feature point in the soft tissue feature point set and the optimized position feature point set to obtain the local structure complexity; Adaptively adjusting the sampling radius according to the local structure complexity, constructing a multi-scale sampling pattern around each feature point, and extracting feature description vectors of each feature point in the soft tissue feature point set and the optimized position feature point set based on the multi-scale sampling pattern; Calculating the mutual information coefficient and the relative position relationship parameter between the feature description vectors, matching the soft tissue feature point set and the optimized position feature point set according to the mutual information coefficient and the relative position relationship parameter, and obtaining an initial matching point pair set; Calculate the mutual information coefficient of the feature description vector and the consistency degree of local space transformation of each pair of matching points in the initial matching point pair set, and generate a matching credibility score in combination with the motion constraints provided by the inertial sensor data; screen the initial matching point pair set according to the matching credibility score, and select the matching point pairs whose matching credibility scores are higher than a preset matching credibility score threshold as the preferred matching point pairs; construct a spatial transformation matrix based on the preferred matching point pairs, and adaptively determine the loss function threshold parameter according to the current matching error distribution; Based on the loss function threshold parameter, combined with the Huber loss function, the parameters of the spatial transformation matrix are optimized, and when the parameter change of the spatial transformation matrix during the optimization process is less than a preset matrix parameter change threshold, the current spatial transformation matrix is ​​determined as the optimal transformation matrix; The soft tissue structure feature data is spatially transformed according to the optimal transformation matrix to generate registration image data.

5. The method according to claim 1, characterized in that Inputting the optimized multi-angle image data and the material characteristic data into the compressed sensing reconstruction model, building regularized constraint conditions in combination with the human soft tissue anatomical structure template, and generating soft tissue density distribution data through iterative calculation includes: Obtaining photon attenuation coefficients according to material characteristic data, constructing a tissue attenuation coefficient matrix, establishing a linear density mapping function based on the tissue attenuation coefficient matrix, and obtaining a material-density mapping model; Constructing an orthogonal sparse basis matrix based on the optimized multi-angle image data, and calculating the sparse representation coefficients of the projection data under the orthogonal sparse basis matrix; Extracting tissue interface position coordinates and interface curvature values ​​from a human soft tissue anatomical structure template, and constructing anatomical feature constraints based on the interface position coordinates and the interface curvature values; Using the interface position coordinates, the density reconstruction space is divided into a plurality of continuous sub-regions, a density continuity constraint is imposed inside the continuous sub-regions, and a piecewise continuous constraint function with controllable jumps is constructed at the interface position coordinates to generate a density distribution regularization term; Calculating structural similarity coefficients of adjacent image blocks in the reconstruction space, constructing a non-local difference operator according to the structural similarity coefficient, applying the non-local difference operator to density distribution data, and establishing a structure preservation constraint term; The material-density mapping model, the anatomical feature constraint, the density distribution regularization term and the structure preservation constraint term are combined into a joint optimization objective function; the joint optimization objective function is decomposed into variables by an augmented Lagrangian method to obtain a projection consistency constraint term and a structure preservation constraint term; Based on the projection consistency constraint, the projection coefficient matrix is ​​solved by the conjugate gradient method, and based on the structure preservation constraint, the constraint parameters are optimized by the proximal gradient descent method to obtain the density distribution image of the current iteration; A three-layer dictionary learning network is constructed based on historical reconstruction data, and feature dictionary groups with different scale organizational structures are obtained through training. Sparsely decompose the density distribution image of the current iteration in the feature dictionary group, calculate the reconstruction error at each scale, select the feature dictionary with the smallest reconstruction error to reconstruct the density distribution, and obtain an optimized density image; The Euclidean distance difference between the optimized density images obtained in two adjacent iterations is calculated, and when the Euclidean distance difference is less than a preset density image distance difference threshold, the current optimized density image is determined as the soft tissue density distribution data.

6. The method according to claim 1, characterized in that Constructing a tissue classification model based on the topological feature data and the morphological feature data, intelligently segmenting different density areas, and generating three-dimensional model data containing tissue layering information includes: Inputting the topological feature data and the morphological feature data into a three-dimensional convolution extraction network to obtain a multi-scale feature mapping sequence, constructing a feature hierarchy network based on the multi-scale feature mapping sequence, calculating a local structure tensor matrix, extracting a tissue boundary direction vector and a principal curvature parameter from the local structure tensor matrix, and generating a structure description vector; Using the structure description vector to construct an organizational connectivity graph model, wherein the graph nodes of the organizational connectivity graph model store the local structure description vector, and the graph edges store the spatial position relationship between adjacent nodes; Constructing a feature enhancement unit based on phase change diffusion, inputting the tissue connectivity graph model into a nonlinear diffusion equation, solving the nonlinear diffusion equation, obtaining enhanced boundary features and enhanced internal structure features, and updating node features of the tissue connectivity graph model; Establish a soft tissue elastic mechanics model based on domain knowledge, convert the tissue stress-strain relationship into node displacement constraint parameters, generate biomechanical constraint conditions, calculate tissue deformation strain energy according to the biomechanical constraint conditions, and construct a regional growth function based on the tissue deformation strain energy minimization criterion; The tissue connectivity graph model is segmented iteratively and hierarchically according to the regional growth function to obtain segmented regions, the values ​​of tissue deformation strain energy of the segmented regions are calculated, and the boundaries where the difference in tissue deformation strain energy of adjacent regions is greater than a preset strain energy difference threshold are determined as tissue demarcation point sets; a tissue hierarchy label set is constructed based on the tissue demarcation point set, a tissue hierarchy rule base is established, and tissue hierarchy data is generated; Performing credibility calculation on the organizational hierarchy data to obtain a regional classification credibility value, and marking the regions whose regional classification credibility values ​​are lower than a preset credibility threshold as a set of regions to be optimized; Adopting a neighborhood information transfer algorithm to iteratively optimize the set of regions to be optimized, updating node feature parameters and edge constraint parameters, until the classification credibility values ​​of all regions are greater than the preset credibility threshold, and obtaining optimized classification identification data; The optimized classification identification data and the density distribution data are spatially registered and fused to generate a three-dimensional soft tissue model with anatomical structure annotations.

7. A multi-angle photography and three-dimensional reconstruction system for soft tissue, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to perform feature analysis on the inspected soft tissue using a deep convolutional neural network to obtain volume parameters and density distribution parameters of the soft tissue; based on the volume parameters and density distribution parameters, generate tube motion trajectory data and sampling interval data through a trajectory optimization algorithm; control a multi-degree-of-freedom manipulator to drive the tube to move according to the tube motion trajectory data, and adjust the multi-degree-of-freedom manipulator to drive the detector assembly to perform synchronous tracking through a visual servo controller; control the tube to collect soft tissue projection images in a preset first energy level range and a second energy level range respectively according to the sampling interval data, and obtain a multi-angle soft tissue projection data set; The second unit is used to input the multi-angle soft tissue projection data set into an image quality assessment network to generate image quality score data, and screen and obtain optimal image sequence data according to the image quality score data; perform material decomposition processing on the optimal image sequence data, and use a multi-scale feature fusion network to extract soft tissue structure feature data and material feature data; perform feature point matching on the soft tissue structure feature data and inertial sensor data, and generate registration image data through a self-correction spatial registration algorithm; and perform adaptive noise reduction processing and local contrast enhancement processing based on wavelet transform on the registration image data in sequence to obtain optimized multi-angle image data; The third unit is used to input the optimized multi-angle image data and the material feature data into a compressed sensing reconstruction model, build regularized constraints in combination with a human soft tissue anatomical structure template, and generate soft tissue density distribution data through iterative calculation; based on the soft tissue density distribution data, an adaptive meshing algorithm is used to perform surface reconstruction to generate initial three-dimensional model data; the initial three-dimensional model data is input into a graph convolutional feature extraction network to extract topological feature data and morphological feature data; a tissue classification model is built based on the topological feature data and the morphological feature data, different density areas are intelligently segmented, and three-dimensional model data containing tissue stratification information is generated.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images