Digital volume correlation method, device, equipment and medium based on deep learning
Directly measuring the deformation field of three-dimensional digital volume images by using deep learning methods, solving the problem of inefficiency of traditional methods, achieving high-precision and high-efficiency real-time three-dimensional deformation field prediction, providing technical support for the fields of materials science and biomedical .
Patent Information
- Application Number
- CN202111575604.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-12-21
AI Technical Summary
Traditional digital volume correlation methods are inefficient due to the iterative calculation process when solving three-dimensional deformation measurements, which limits the realization of real-time measurements.
Using deep learning-based methods, by constructing deep learning networks, including holistic pixel-related networks, subvome-related networks and displacement optimization networks, the deformation field measurement of three-dimensional digital volume images is directly carried out to avoid iterative solutions, and training with three-dimensional image datasets Network to achieve three-dimensional deformation field prediction with subvoxel accuracy.
It improves measurement efficiency and accuracy, realizes real-time subvampus accuracy three-dimensional deformation field prediction, and provides technical support for real-time three-dimensional deformation measurement in fields such as materials science and biomedicine.
Smart Images

Figure CN114549614B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of optical measurement technology, and in particular to a digital volume correlation method, device, equipment and medium based on deep learning. Background Art
[0002] Accurate quantification of three-dimensional deformation fields plays an important role in aerospace, materials science, biomedicine, engineering and construction, etc. Different from contact displacement measurement based on sensors and other methods, optical measurement methods can obtain full-field deformation without direct contact with the test piece, which will not damage the test piece and can adapt to various measurement scales and measurement conditions, so it has a wider range of application scenarios.
[0003] The Digital Volume Correlation (DVC) method is a typical method in optical measurement. It determines the displacement value of each sampling point by calculating the correlation of the digital volume image before and after deformation. It is a generalization of the two-dimensional Digital Image Correlation (DIC) method to three-dimensional problems. The characterization results of the digital volume correlation method for displacement are more accurate and efficient than other optical measurement methods, and have now been successfully applied in a series of applications.
[0004] The specific implementation process of the traditional digital volume correlation method is as follows: first, the position of interest is selected as the sampling point in the three-dimensional digital volume image, and the sub-area is divided in the image before deformation with this as the center for subsequent correlation calculation; secondly, each sampling point performs a whole-voxel search calculation in the deformed image, and determines the position with the largest correlation coefficient in a point-by-point scanning manner, and then obtains the corresponding whole-voxel displacement value; finally, based on the whole-voxel displacement, the displacement distribution in the sub-area is approximated by the affine shape function, the sub-area of the image before and after deformation is matched, and the final displacement value is determined by iteration, thereby realizing displacement measurement with sub-voxel accuracy.
[0005] However, in order to improve the accuracy of deformation measurement, the traditional digital volume correlation method determines the final displacement value through iteration. This iterative solution calculation process seriously reduces the measurement efficiency of the digital volume correlation method, limits the number of sampling points, and hinders the realization of real-time 3D deformation measurement. Summary of the invention
[0006] 1. Technical issues to be resolved
[0007] In response to the current technical problems, the present disclosure proposes a digital volume correlation method, device, equipment and medium based on deep learning.
[0008] (II) Technical solution
[0009] In order to solve the above technical problems, the technical solutions adopted by the present disclosure are as follows:
[0010] The present disclosure provides a digital volume correlation method based on deep learning, comprising: generating a three-dimensional image data set for deep learning network training; constructing a deep learning network; training the deep learning network using the three-dimensional image data set; inputting three-dimensional digital volume images before and after deformation into the trained deep learning network and performing calculations to obtain a three-dimensional deformation field with sub-voxel accuracy.
[0011] In the above scheme, generating a three-dimensional image dataset for deep learning network training includes: generating a series of simulated three-dimensional digital volume images before deformation and simulated three-dimensional digital volume images after deformation as datasets for deep learning network training and verification.
[0012] In the above scheme, the gray value G0 of any point (X, Y, Z) in the simulated three-dimensional digital volume image before deformation is determined by the following formula:
[0013]
[0014] where a x 、a y 、a z are the full width at half maximum of the point spread function along the x, y, and z directions, respectively, (X c,i , Y c,i , Z c,i ) is the center position of the i-th speckle particle, b is the background intensity of the three-dimensional simulated image, and n is the number of speckle particles.
[0015] In the above scheme, the gray value G1 of any point (x, y, z) in the deformed simulated three-dimensional digital volume image is determined by the following formula:
[0016]
[0017] Among them, u c,i 、v c,i 、w c,i They are points (X c,i , Y c,i , Z c,i ) are the displacement components along the x, y, and z directions.
[0018] In the above scheme, in the step of constructing a deep learning network, the deep learning network includes the following three networks that perform different functions:
[0019] A global pixel correlation network is used to calculate the global pixel displacement of each sampling point in the three-dimensional digital volume image;
[0020] A subvoxel correlation network is used to calculate the subvoxel displacement of each sampling point in a three-dimensional digital volume image;
[0021] The displacement optimization network is used to perform smoothing and denoising post-processing on the 3D displacement field output by the subvoxel correlation network.
[0022] In the above scheme, the whole pixel correlation network includes a whole pixel image segmentation layer, a first whole pixel three-dimensional convolution layer 11, a first whole pixel three-dimensional maximum pooling layer 21, a second whole pixel three-dimensional convolution layer 12, a second whole pixel three-dimensional maximum pooling layer 22, a third whole pixel three-dimensional convolution layer 13, a third whole pixel three-dimensional maximum pooling layer 23 and a whole pixel fully connected layer, and the whole pixel correlation network calculates the whole pixel displacement of each sampling point in the three-dimensional digital volume image, including:
[0023] First, the input 3D digital volume image before deformation is split into M pixels with sampling points (X s , Y s , Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first whole pixel three-dimensional convolution layer 11 together with the corresponding deformed sub-image; wherein the corresponding deformed sub-image is based on the sampling point (x s ,y s , z s ) as the center, with (D x , D y , D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s , z s ) can be obtained by (X s , Y s , Z s ) and displacement correction term (u s , v s , w s )Sure:
[0024] x s =X s +u s
[0025] y s =Y s +v s
[0026] z s =Z s +w s
[0027] In the overall voxel correlation network, the displacement correction term is taken as u s =v s =w s =0;
[0028] Secondly, in the first whole pixel three-dimensional convolution layer 11, the input three-dimensional sub-image before and after deformation is subjected to a three-dimensional convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function;
[0029] Subsequently, the feature map extracted by the convolution operation is subjected to a three-dimensional maximum pooling operation in the first whole-pixel three-dimensional maximum pooling layer 21 to extract the main features;
[0030] Next, the second whole pixel 3D convolution layer 12, the second whole pixel 3D maximum pooling layer 22, the third whole pixel 3D convolution layer 13 and the third whole pixel 3D maximum pooling layer 23 in the whole pixel correlation network further abstract the feature information learned in the previous layer in turn, and finally integrate them through the whole pixel fully connected layer to obtain an output of a specified size (1×1×1×3×M), which is the three displacement components of the whole pixel accuracy of each sampling point.
[0031] In the above scheme, the subvoxel correlation network includes a subvoxel image segmentation layer, a first subvoxel three-dimensional convolution layer 31, a first subvoxel three-dimensional maximum pooling layer 41, a second subvoxel three-dimensional convolution layer 32, a second subvoxel three-dimensional maximum pooling layer 42, a third subvoxel three-dimensional convolution layer 33, a third subvoxel three-dimensional maximum pooling layer 43 and a subvoxel fully connected layer, and the subvoxel correlation network calculates the subvoxel displacement of each sampling point in the three-dimensional digital volume image, including:
[0032] First, the input 3D digital image before deformation is split into M subvoxel image segmentation layers with sampling points (X s , Y s , Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first sub-voxel three-dimensional convolution layer 31 together with the corresponding deformed sub-image; wherein the corresponding deformed sub-image is based on the sampling point (x s ,y s , z s ) as the center, with (D x , D y , D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s , z s ) can be obtained by (X s , Ys , Z s ) and displacement correction term (u s , v s , w s )Sure:
[0033] x s =X s +u s
[0034] y s =Y s +v s
[0035] z s =Z s +w s
[0036] In the subvoxel correlation network, the displacement correction term is taken as the output of the overall voxel correlation network;
[0037] Secondly, in the first subvoxel 3D convolution layer 31, the input 3D sub-images before and after deformation are subjected to 3D convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function;
[0038] Subsequently, the feature map extracted by the convolution operation is subjected to a 3D maximum pooling operation in the first sub-voxel 3D maximum pooling layer 41 to extract main features;
[0039] Next, the second subvoxel 3D convolution layer 32, the second subvoxel 3D maximum pooling layer 42, the third subvoxel 3D convolution layer 33 and the third subvoxel 3D maximum pooling layer 43 in the subvoxel correlation network further abstract the feature information learned in the previous layer, and finally integrate them through the subvoxel fully connected layer to obtain an output of a specified size (1×1×1×3×M), which is the three displacement components of the subvoxel accuracy of each sampling point.
[0040] In the above scheme, the displacement optimization network includes a local displacement assembly layer, a displacement optimization fully connected layer and a global displacement assembly layer. The displacement optimization network performs smooth denoising post-processing on the three-dimensional displacement field output by the subvoxel correlation network, including: first, the local displacement assembly layer integrates the three-dimensional displacement fields of each input sampling point to obtain integrated spatial information; secondly, the displacement optimization fully connected layer determines the least squares coefficient of the local displacement based on the displacement field after integrating the spatial information; finally, the global displacement assembly layer outputs the final three-dimensional full-field displacement distribution according to the determined least squares coefficient.
[0041] In the above solution, in the step of integrating the input three-dimensional displacement fields of the sampling points by the local displacement assembly layer, the local displacement assembly layer integrates the displacement information of five adjacent sampling points.
[0042] In the above scheme, the step of training the deep learning network using the three-dimensional image data set is to optimize the parameters to be learned in each layer of the deep learning network using the three-dimensional image data set.
[0043] In the above scheme, the use of the three-dimensional image data set to optimize the parameters to be learned in each layer of the deep learning network specifically includes: using 80% of the three-dimensional image data to train the deep learning network, using 15% of the three-dimensional image data to verify the deep learning network, and using 5% of the three-dimensional image data to test the deep learning network; wherein, the network hyperparameters are determined by random search, the root mean square error is specified as the loss function, the Adam optimizer is used to optimize the loss, and the iteration is terminated when the loss value is less than the specified threshold or the number of iterations is greater than the specified value, and the parameter values of each layer of the deep learning network are saved for subsequent prediction of the target displacement field.
[0044] In the above scheme, the three-dimensional digital volume images before and after deformation are input to the trained deep learning network and calculation is performed to obtain a three-dimensional deformation field with sub-voxel accuracy, including: the three-dimensional digital volume images before and after deformation to be tested are input to the trained deep learning network; this group of three-dimensional digital volume images to be tested are first mapped by each layer of the whole voxel correlation network to obtain a three-dimensional displacement with whole voxel accuracy and used as a displacement correction of the sub-voxel correlation network; then, the three-dimensional digital volume images before and after deformation are mapped by each layer of the sub-voxel correlation network to obtain a three-dimensional displacement with sub-voxel accuracy; finally, a smooth three-dimensional deformation field is output by the displacement optimization network.
[0045] In the above scheme, the trained deep learning network takes the three-dimensional digital volume images before and after deformation as input, and takes the smooth sub-voxel accurate full-field deformation as output.
[0046] Another aspect of the present disclosure provides a digital volume correlation device based on deep learning, comprising:
[0047] A three-dimensional image data set generation module, used to generate a three-dimensional image data set for deep learning network training;
[0048] Deep learning network building module, used to build deep learning networks;
[0049] A deep learning network training module, used for training a deep learning network using a three-dimensional image dataset; and
[0050] The target displacement field prediction module is used to input the three-dimensional digital volume images before and after deformation into the trained deep learning network and perform calculations to obtain the three-dimensional deformation field with sub-voxel accuracy.
[0051] On the other hand, the present disclosure provides an electronic device, including: a processor; and a memory storing a computer executable program, which, when executed by the processor, enables the processor to perform the deep learning-based digital volume correlation method.
[0052] In yet another aspect, the present disclosure provides a storage medium comprising computer executable instructions, which, when executed, implement the deep learning-based digital volume correlation method.
[0053] In yet another aspect, the present disclosure provides a computer program, comprising: computer executable instructions, which are used to implement the deep learning-based digital volume correlation method when executed.
[0054] (III) Beneficial effects
[0055] The deep learning-based digital volume correlation method, device, equipment and medium provided by the present invention introduce a deep learning network into the digital volume correlation method for the first time, and use the deep learning network to directly measure the deformation field of the three-dimensional digital volume image, thereby avoiding complex iterative solutions, simplifying the calculation, and having high measurement accuracy while also having extremely high measurement efficiency. It can realize real-time sub-voxel accuracy three-dimensional deformation field prediction, and provides strong technical support for real-time three-dimensional deformation measurement in the fields of material science, biomedicine, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more fully understand the present disclosure and its advantages, a detailed description of the accompanying drawings is now given, in which:
[0057] Figure 1 is a flowchart of a digital volume correlation method based on deep learning according to an embodiment of the present disclosure.
[0058] Figure 2 It is a structural schematic diagram of a deep learning network in a digital volume correlation method based on deep learning according to an embodiment of the present disclosure.
[0059] Figure 3A It is a structural schematic diagram of the overall voxel correlation network in the deep learning-based digital volume correlation method according to an embodiment of the present disclosure.
[0060] Figure 3B It is a structural schematic diagram of a subvoxel correlation network in a deep learning-based digital volume correlation method according to an embodiment of the present disclosure.
[0061] Figure 3C It is a structural schematic diagram of a displacement optimization network in a digital volume correlation method based on deep learning according to an embodiment of the present disclosure.
[0062] Figure 4AThis is an image of the three-dimensional digital volume image at the Z=120 voxel section before deformation in Embodiment 1 of the present disclosure, and the white box in the figure shows the area to be calculated.
[0063] Figure 4B It is an image of the deformed three-dimensional digital volume image at the Z=120 voxel section in Embodiment 1 of the present disclosure.
[0064] Figure 5A : is a distribution cloud diagram of the component of the theoretical deformation field along the x direction at the Z=120 voxel section in Example 1 of the present disclosure.
[0065] Figure 5B It is a distribution cloud diagram of the component of the deformation field along the x direction at the Z=120 voxel section calculated using the traditional digital volume correlation method in Example 1 of the present disclosure.
[0066] Figure 5C 1 is a distribution cloud diagram of the component of the deformation field along the x direction at the Z=120 voxel section calculated using the method of the present disclosure in Example 1 of the present disclosure.
[0067] Figure 5D : is a distribution cloud diagram of the component of the theoretical deformation field along the y direction at the Z=120 voxel section in Example 1 of the present disclosure.
[0068] Figure 5E It is a distribution cloud diagram of the component of the deformation field along the y direction at the Z=120 voxel section calculated using the traditional digital volume correlation method in Example 1 of the present disclosure.
[0069] Fig. 5F 1 is a distribution cloud diagram of the component of the deformation field along the y direction at the Z=120 voxel section calculated using the method of the present disclosure in Example 1 of the present disclosure.
[0070] Figure 6 The figure shows the comparison between the displacement field measurement error and the calculation time of each voxel point of the deep learning-based digital volume correlation method according to the embodiment of the present disclosure and the traditional digital volume correlation method.
[0071] Figure 7 A block diagram of a deep learning-based digital volume correlation device according to an embodiment of the present disclosure is schematically shown.
[0072] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown.
[0073] [Reference Signs]
[0074] 700: Digital volume correlation device based on deep learning
[0075] 701: 3D image dataset generation module
[0076] 702: Deep Learning Network Building Blocks
[0077] 703: Deep Learning Network Training Module
[0078] 704: Target displacement field prediction module
[0079] 800: Electronic equipment
[0080] 810: Processor
[0081] 820: Memory
[0082] 821: Computer Programs DETAILED DESCRIPTION
[0083] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0084] The embodiments of the present disclosure provide a digital volume correlation method, device, equipment and medium based on deep learning. Figure 1 is a flow chart of a digital volume correlation method based on deep learning according to an embodiment of the present disclosure. It should be noted that Figure 1 The examples shown are merely examples of application scenarios to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other environments or scenarios.
[0085] like Figure 1 As shown, the digital volume correlation method based on deep learning in the embodiment of the present disclosure includes the following steps:
[0086] Step S1: Generate a three-dimensional image dataset for deep learning network training.
[0087] In this step, a series of simulated three-dimensional digital volume images before deformation and simulated three-dimensional digital volume images after deformation are generated as data sets for deep learning network training and verification. The gray value G0 of any point (X, Y, Z) in the simulated three-dimensional digital volume image before deformation can be determined by the following formula:
[0088]
[0089] where a x 、ay 、a z are the full width at half maximum of the point spread function along the x, y, and z directions, respectively, (X c,i , Y c,i , Z c,i ) is the center position of the i-th speckle particle, b is the background intensity of the three-dimensional simulated image, and n is the number of speckle particles.
[0090] Accordingly, the gray value G1 of any point (x, y, z) in the deformed simulated three-dimensional digital volume image is determined by the following formula:
[0091]
[0092] Among them, u c,i 、v c,i 、w c,i They are points (X c,i , Y c,i , Z c,i ) are the displacement components along the x, y, and z directions.
[0093] In the present disclosure, u, v, and w are all distributed in the form of a quadratic function of position (X, Y, Z), that is:
[0094] u=α1x 2 +α2y 2 +α3z 2 +α4xy+α5yz+α6zx+α7x+α8y+α9z+α 10
[0095] v=β1x 2 +β2y 2 +β3z 2 +β4xy+β5yz+β6zx+β7x+β8y+β9z+β 10
[0096] w=γ1x 2 +γ2y 2 +γ3z 2 +γ4xy+γ5yz+γ6zx+γ7x+γ8y+γ9z+γ 10
[0097] where α j , β j , γ jis a randomly selected coefficient, j = 1, 2, ..., 10. Since the subvoxel correlation network performs deformation measurement with subvoxel accuracy on the basis of the overall voxel correlation network, the displacement field in the data set for training the subvoxel correlation network satisfies u≤1 voxel, v≤1 voxel and w≤1 voxel. Finally, the random 3D displacement field and the corresponding 3D displacement field after least squares fitting are used as the training input and label of the displacement optimization network, respectively, for training the network in subsequent steps.
[0098] Step S2: construct a deep learning network;
[0099] In this step, if Figure 2 The deep learning network model shown is established, which includes the following three networks that perform different functions: an overall voxel correlation network, a subvoxel correlation network, and a displacement optimization network.
[0100] Among them, the whole pixel correlation network is used to calculate the whole pixel displacement of each sampling point in the three-dimensional digital volume image. Its specific network structure is as follows: Figure 3A As shown, it includes a whole pixel image segmentation layer, a first whole pixel three-dimensional convolution layer 11, a first whole pixel three-dimensional maximum pooling layer 21, a second whole pixel three-dimensional convolution layer 12, a second whole pixel three-dimensional maximum pooling layer 22, a third whole pixel three-dimensional convolution layer 13, a third whole pixel three-dimensional maximum pooling layer 23 and a whole pixel fully connected layer, and the whole pixel correlation network calculates the whole pixel displacement of each sampling point in the three-dimensional digital volume image, including:
[0101] First, in the overall pixel image segmentation layer, the input 3D digital volume image before deformation is split into M pixels with sampling points (X s , Y s , Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first whole pixel three-dimensional convolution layer 11 together with the corresponding deformed sub-image; wherein the corresponding deformed sub-image is based on the sampling point (x s ,y s , z s ) as the center, with (D x , D y , D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s , z s ) can be obtained by (X s , Y s , Z s ) and displacement correction term (u s , v s , ws )Sure:
[0102] x s =X s +u s
[0103] y s =Y s +v s
[0104] z s =Z s +w s
[0105] In the overall voxel correlation network, the displacement correction term is taken as u s =v s =w s =0.
[0106] Secondly, in the first whole pixel 3D convolution layer 11, the input 3D sub-image before and after deformation is subjected to a 3D convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function; subsequently, the feature map extracted by the convolution operation is subjected to a 3D maximum pooling operation in the first whole pixel 3D maximum pooling layer 21 to extract the main features;
[0107] Next, the other components in the whole pixel correlation network, namely the second whole pixel 3D convolution layer 12, the second whole pixel 3D maximum pooling layer 22, the third whole pixel 3D convolution layer 13 and the third whole pixel 3D maximum pooling layer 23, further abstract the feature information learned in the previous layer in turn, and finally integrate them through the whole pixel fully connected layer to obtain an output of a specified size (1×1×1×3×M), which is the three displacement components of the whole pixel accuracy of each sampling point.
[0108] The subvoxel correlation network is used to calculate the subvoxel displacement of each sampling point in the three-dimensional digital volume image. The specific network structure is as follows: Figure 3B As shown, it includes a subvoxel image segmentation layer, a first subvoxel three-dimensional convolution layer 31, a first subvoxel three-dimensional maximum pooling layer 41, a second subvoxel three-dimensional convolution layer 32, a second subvoxel three-dimensional maximum pooling layer 42, a third subvoxel three-dimensional convolution layer 33, a third subvoxel three-dimensional maximum pooling layer 43 and a subvoxel fully connected layer. The subvoxel correlation network calculates the subvoxel displacement of each sampling point in the three-dimensional digital volume image, including:
[0109] First, the input 3D digital image before deformation is split into M subvoxel image segmentation layers with sampling points (X s , Y s , Z s ) is the center, and the dimensions in the x, y, and z directions are D x, D y , D z The sub-image is input into the first sub-voxel three-dimensional convolution layer 31 together with the corresponding deformed sub-image; wherein the corresponding deformed sub-image is based on the sampling point (x s ,y s , z s ) as the center, with (D x , D y , D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s , z s ) can be obtained by (X s , Y s , Z s ) and displacement correction term (u s , v s , w s )Sure:
[0110] x s =X s +u s
[0111] y s =Y s +v s
[0112] z s =Z s +w s
[0113] In the subvoxel correlation network, the displacement correction term is taken as the output of the overall voxel correlation network.
[0114] Secondly, in the first subvoxel 3D convolution layer 31, the input 3D sub-image before and after deformation is subjected to 3D convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function. Subsequently, the feature map extracted by the convolution operation is subjected to 3D maximum pooling operation in the first subvoxel 3D maximum pooling layer 41 to extract the main features. Next, the other components in the subvoxel correlation network, namely the second subvoxel 3D convolution layer 32, the second subvoxel 3D maximum pooling layer 42, the third subvoxel 3D convolution layer 33 and the third subvoxel 3D maximum pooling layer 43, further abstract the feature information learned in the previous layer in turn, and finally integrate through the subvoxel fully connected layer to obtain an output of a specified size (1×1×1×3×M), which is the three displacement components of the subvoxel accuracy of each sampling point.
[0115] The displacement optimization network is used to perform smooth denoising on the three-dimensional displacement field output by the subvoxel correlation network. The specific network structure is as follows: Figure 3CAs shown, it includes a local displacement assembly layer, a displacement optimization fully connected layer and a global displacement assembly layer. The displacement optimization network performs smooth denoising post-processing on the three-dimensional displacement field output by the subvoxel correlation network, including:
[0116] First, the local displacement assembly layer integrates the three-dimensional displacement field of each input sampling point to obtain integrated spatial information; considering the continuity between the displacement of each sampling point and its adjacent sampling points, the local displacement assembly layer in this disclosure integrates the displacement information of the five adjacent sampling points. Secondly, the displacement optimization fully connected layer determines the least squares coefficient of the local displacement based on the displacement field after integrating the spatial information. Finally, the global displacement assembly layer outputs the final three-dimensional full-field displacement distribution based on the determined least squares coefficient.
[0117] In the embodiment of the present disclosure, step S1 is to generate a three-dimensional image data set for deep learning network training, and step S2 is to build a deep learning network, that is, first generate a three-dimensional image data set for deep learning network training, and then build the deep learning network. In practical applications, the deep learning network can also be built first, and then the three-dimensional image data set for deep learning network training can be generated. This is consistent with the technical thinking of the present disclosure and should be included in the protection scope of the present disclosure.
[0118] Step S3: training the deep learning network using the three-dimensional image dataset;
[0119] In this step, the use of the three-dimensional image data set to train the deep learning network is to use the three-dimensional image data set to optimize the parameters to be learned in each layer of the deep learning network.
[0120] In the present disclosure, 80% of the three-dimensional image data is used to train the deep learning network, 15% of the three-dimensional image data is used to verify the deep learning network, and 5% of the three-dimensional image data is used to test the deep learning network, wherein the network hyperparameters are determined by random search, the root mean square error is specified as the loss function, the Adam optimizer is used to optimize the loss, and the iteration is terminated when the loss value is less than a specified threshold or the number of iterations is greater than a specified value, and the parameter values of each layer of the deep learning network are saved for subsequent prediction of the target displacement field.
[0121] Step S4: Target displacement field prediction, that is, inputting the three-dimensional digital volume images before and after deformation into the trained deep learning network and performing calculations to obtain a three-dimensional deformation field with sub-voxel accuracy.
[0122] In this step, the three-dimensional digital volume images before and after deformation are input into the trained deep learning network; the three-dimensional digital volume images to be tested are first mapped by the layers of the whole voxel correlation network to obtain the three-dimensional displacement with the whole voxel accuracy and used as the displacement correction of the subvoxel correlation network; then, the three-dimensional digital volume images before and after deformation are mapped by the layers of the subvoxel correlation network to obtain the three-dimensional displacement with the subvoxel accuracy; finally, the smooth three-dimensional deformation field is output by the displacement optimization network. Among them, the trained deep learning network takes the three-dimensional digital volume images before and after deformation as input, and takes the smooth subvoxel accuracy full-field deformation as output. At this time, the parameters of each layer of the deep learning network have been determined in the deep learning network training step, so the mapping of the data by each layer is also known, and the smooth three-dimensional displacement field with the subvoxel accuracy can be obtained efficiently.
[0123] Example 1
[0124] In Example 1 of the present disclosure, a single sinusoidal displacement field with periodic variation is used as an example to predict it using the method of the present disclosure, so as to further illustrate the implementation process of the digital volume correlation method based on deep learning provided by the present disclosure. The specific steps of using the method of the present disclosure are as follows:
[0125] First: 3D image dataset generation;
[0126] In this step, a series of simulated 3D speckle images before and after deformation are generated as a data set for network training and verification. The gray value G0 of any point (X, Y, Z) in the image before deformation can be determined by the following formula:
[0127]
[0128] where a x 、a y 、a z are the full width at half maximum of the point spread function along the x, y, and z directions, and in this embodiment are respectively taken as a x = 0.63 voxels, a y = 0.63 voxels, a z =0.63 voxels. (X c,i , Y c,i , Z c,i ) is the center position of the i-th speckle particle, which is obtained by random generation. b is the background intensity of the three-dimensional simulated image, which is taken as b=1 in this embodiment. n is the number of speckle particles, which is taken as n=90 in this embodiment. The gray value G1 of any point (x, y, z) in the corresponding deformed image can be written as:
[0129]
[0130] where uc,i 、v c,i 、w c,i They are points (X c,i , Y c,i , Z c,i ) are displacement components along the x, y, and z directions. In the present disclosure, u, v, and w are distributed in the form of a quadratic function of position (X, Y, Z), that is:
[0131] u=α1x 2 +α2y 2 +α3z 2 +α4xy+α5yz+α6zx+α7x+α8y+α9z+α 10
[0132] v=β1x 2 +β2y 2 +β3z 2 +β4xy+β5yz+β6zx+β7x+β8y+β9z+β 10
[0133] w=γ1x 2 +γ2y 2 +γ3z 2 +γ4xy+γ5yz+γ6zx+γ7x+γ8y+γ9z+γ 10
[0134] where α j , β j , γ j is a randomly selected coefficient, j = 1, 2, ..., 10. Since the subvoxel correlation network performs deformation measurement with subvoxel accuracy on the basis of the overall voxel correlation network, the displacement field in the data set for training the subvoxel correlation network satisfies u≤1 voxel, v≤1 voxel and w≤1 voxel. Finally, the random three-dimensional displacement field and the corresponding three-dimensional displacement field after least squares fitting are used as the training input and label of the displacement optimization network, respectively, for training the network in subsequent steps. In this embodiment, the sample size of the data set is 40,000.
[0135] Second: deep learning network construction;
[0136] In this step, if Figure 2 The deep learning network model shown is established. The deep learning model consists of three networks with different functions, namely: the whole voxel correlation network, the subvoxel correlation network and the displacement optimization network.
[0137] The whole pixel correlation network is used to calculate the whole pixel displacement of each sampling point in the image. Its specific network structure is as follows: Figure 3AAs shown, it includes a whole pixel image segmentation layer, a first whole pixel 3D convolution layer 11, a first whole pixel 3D maximum pooling layer 21, a second whole pixel 3D convolution layer 12, a second whole pixel 3D maximum pooling layer 22, a third whole pixel 3D convolution layer 13, a third whole pixel 3D maximum pooling layer 23 and a whole pixel fully connected layer. First, in the whole pixel image segmentation layer, the input three-dimensional digital image before deformation is split into M sampling points (X s , Y s , Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first whole pixel three-dimensional convolution layer 11 together with the corresponding deformed sub-image. The corresponding deformed sub-image is based on the sampling point (x s ,y s , z s ) as the center, with (D x , D y , D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s , z s ) can be obtained by (X s , Y s , Z s ) and displacement correction term (u s , v s , w s )Sure:
[0138] x s =X s +u s
[0139] y s =Y s +v s
[0140] z s =Z s +w s
[0141] In the overall voxel correlation network, the displacement correction term is taken as u s =v s =w s =0. In this embodiment, take D x =D y =D z=21 voxels. Secondly, in the first whole-pixel 3D convolution layer 11, the input 3D sub-image before and after deformation is subjected to 3D convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function. Subsequently, the feature map extracted by the convolution operation is subjected to 3D maximum pooling operation in the first whole-pixel 3D maximum pooling layer 21 to extract the main features. The other components in the whole-pixel correlation network, namely the second whole-pixel 3D convolution layer 12, the second whole-pixel 3D maximum pooling layer 22, the third whole-pixel 3D convolution layer 13 and the third whole-pixel 3D maximum pooling layer 23, similarly further abstract the feature information learned in the previous layer in turn, and finally integrate through the whole-pixel fully connected layer to obtain an output tensor of a specified size (1×1×1×3×M). As described in the step of generating a 3D image dataset, the output is the three displacement components of the whole-pixel accuracy of each sampling point.
[0142] The subvoxel correlation network is used to calculate the subvoxel displacement of each sampling point in the image. Its specific network structure is as follows: Figure 3B As shown, the system includes a subvoxel image segmentation layer, a first subvoxel 3D convolution layer 31, a first subvoxel 3D maximum pooling layer 41, a second subvoxel 3D convolution layer 32, a second subvoxel 3D maximum pooling layer 42, a third subvoxel 3D convolution layer 33, a third subvoxel 3D maximum pooling layer 43 and a subvoxel fully connected layer. First, the input 3D digital image before deformation is split into M subvoxel images with sampling points (X s , Y s , Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first sub-voxel 3D convolution layer 31 together with the corresponding deformed sub-image. The corresponding deformed sub-image is based on the sampling point (x s ,y s , z s ) as the center, with (D x , D y , D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s , z s ) can be obtained by (X s , Y s , Z s ) and displacement correction term (u s , v s , w s )Sure:
[0143] x s =X s +us
[0144] y s =Y s +v s
[0145] z s =Z s +w s
[0146] In the subvoxel correlation network, the displacement correction term is taken as the output of the overall voxel correlation network. Secondly, in the first subvoxel 3D convolution layer 31, the input 3D sub-image before and after deformation is subjected to a 3D convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function. Subsequently, the feature map extracted by the convolution operation is subjected to a 3D maximum pooling operation in the first subvoxel 3D maximum pooling layer 41 to extract the main features. The other components in the subvoxel correlation network, namely the second subvoxel 3D convolution layer 32, the second subvoxel 3D maximum pooling layer 42, the third subvoxel 3D convolution layer 33 and the third subvoxel 3D maximum pooling layer 43, similarly further abstract the feature information learned in the previous layer in turn, and finally integrate through the subvoxel fully connected layer to obtain an output tensor of a specified size (1×1×1×3×M). As described in the step of generating a 3D image dataset, the output is the three displacement components of the subvoxel accuracy of each sampling point.
[0147] The displacement optimization network is used to perform smooth denoising on the three-dimensional displacement field output by the subvoxel network. The specific network structure is as follows: Figure 3C As shown, it includes a local displacement assembly layer, a displacement optimization fully connected layer and a global displacement assembly layer. The three-dimensional displacement values of each sampling point input are first integrated with spatial information in the local displacement assembly layer. Considering the continuity between the displacement of each sampling point and its adjacent sampling points, the local displacement assembly layer in the present disclosure integrates the displacement information of the five adjacent sampling points. Secondly, the displacement optimization fully connected layer is specified, and the displacement optimization fully connected layer determines the least squares coefficient of the local displacement based on the displacement field after integrating the spatial information. Finally, the global displacement assembly layer can easily output the final three-dimensional full-field displacement distribution according to the determined least squares coefficient.
[0148] In the disclosed embodiment, the number of convolution kernels of the first whole-voxel 3D convolution layer 11 and the first subvoxel 3D convolution layer 31 is 32, the number of convolution kernels of the second whole-voxel 3D convolution layer 12 and the second subvoxel 3D convolution layer 32 is 64, and the number of convolution kernels of the third whole-voxel 3D convolution layer 13 and the third subvoxel 3D convolution layer 33 is 128, so as to gradually extract deep information. In addition, the convolution kernel size of each convolution layer is 5 voxels, and the pooling window size of the maximum pooling layer is 3 voxels.
[0149] Third: deep learning network training;
[0150] In this step, the data set generated by the three-dimensional image data set generation step is used to optimize the parameters to be learned in each layer of the deep learning network. In the present disclosure, 80% of the data in the data set is used to train the deep learning network, 15% of the data is used to verify the deep learning network, and 5% of the data is used to test the deep learning network. The network hyperparameters are determined by random search. The root mean square error is specified as the loss function, and the Adam optimizer is used to optimize the loss. When the loss function is less than the specified threshold or the number of iterations is greater than the specified value, the iteration terminates, and the parameter values of each layer of the deep learning network are saved for subsequent prediction of the target displacement field. In the current embodiment, the learning rate of the training is taken as 0.00001, the loss threshold is specified as 0.0001, and the maximum number of iterations is specified as 30,000 times.
[0151] Fourth: target displacement field prediction.
[0152] In this step, the digital volume images before and after the three-dimensional deformation to be measured are input into the trained deep learning network.
[0153] In the current embodiment, the three-dimensional deformation field to be measured is distributed in a single sinusoidal form:
[0154] u0=η x sin(2πx / τ x )
[0155] v0=η y sin(2πy / τ y )
[0156] w0=η z sin(2πz / τ z )
[0157] Among them, η x , η y , η z Take η as x =η y =η z = 1.2 voxels, τ x , τ y , τ z Take τ as x = 80 voxels, τ y = 90 voxels and τ z =100 voxels. The size of the three-dimensional digital volume image before and after deformation is 400 voxels × 400 voxels × 400 voxels, and the gray value of each voxel point is given by the following formula:
[0158]
[0159]
[0160] and
[0161] x c,i =X c,i +u0(X c,i , Y c,i , Z c,i )
[0162] y c,i =Y c,i +v0(X c,i , Y c,i , Z c,i )
[0163] z c,i =Z c,i +w0(X c,i , Y c,i , Z c,i )
[0164] Among them, a x = 0.63 voxels, a y = 0.63 voxels, a z = 0.63 voxels, n = 12800000. The three-dimensional digital volume images before and after deformation at the Z = 120 voxel section are respectively Figure 4A and Figure 4B Given in Figure 4A is an image of the three-dimensional digital volume image before deformation at the voxel section Z=120 in Embodiment 1 of the present disclosure, and the area to be calculated is Figure 4A Indicated by white box; Figure 4B It is an image of the deformed three-dimensional digital volume image at the voxel section Z=120 in Embodiment 1 of the present disclosure.
[0165] This set of images to be tested is first mapped by each layer of the whole voxel correlation network to obtain the three-dimensional displacement with whole voxel accuracy and used as the displacement correction of the subvoxel correlation network. Subsequently, the three-dimensional digital volume images before and after deformation are mapped by each layer of the subvoxel correlation network to obtain the three-dimensional displacement with subvoxel accuracy. Finally, the smooth three-dimensional deformation field is output by the displacement optimization network. At this time, the parameters of each layer of the deep learning network have been determined in the deep learning network training step, so the mapping of the data by each layer is also known, and the smooth three-dimensional displacement field with subvoxel accuracy can be obtained efficiently.
[0166] FIG5 shows a comparison of the distribution cloud diagrams of the displacement field in the current embodiment. Figure 5A It is a distribution cloud diagram of the component of the theoretical deformation field along the x direction at the Z=120 voxel section in this embodiment. Figure 5B1 is a distribution cloud diagram of the component of the deformation field along the x direction at the Z=120 voxel section calculated using the traditional digital volume correlation method in this embodiment. Figure 5C It is a distribution cloud diagram of the component of the deformation field along the x direction at the Z=120 voxel section calculated by the method disclosed in this embodiment. Figure 5D It is a distribution cloud diagram of the component of the theoretical deformation field along the y direction at the Z=120 voxel section in this embodiment. Figure 5E 1 is a distribution cloud diagram of the component of the deformation field along the y direction at the Z=120 voxel section calculated using the traditional digital volume correlation method in this embodiment. Fig. 5F : is a distribution cloud diagram of the component of the deformation field along the y direction at the Z=120 voxel section calculated using the disclosed method in this embodiment.
[0167] Figure 6 The comparison of the deformation field calculation error of the digital volume correlation method based on deep learning of the embodiment of the present disclosure and the deformation field calculation error of the traditional digital volume correlation method is shown, wherein Δ|u-u0|, Δ|v-v0| and Δ|w-w0| are the mean values of the errors between the displacement field calculated along the x, y, and z directions and the theoretical displacement field, and σ|u-u0|, σ|v-v0| and σ|w-w0| are the standard deviations of the errors between the displacement field calculated along the x, y, and z directions and the theoretical displacement field. It can be seen that the calculation accuracy of the digital volume correlation method based on deep learning of the present disclosure is higher than that of the traditional digital volume correlation method.
[0168] at the same time, Figure 6 The figure also shows the comparison between the calculation time of each voxel point by the deep learning-based digital volume correlation method of the embodiment of the present disclosure and the calculation time of each voxel point by the deformation field of the traditional digital volume correlation method. It can be seen that the calculation efficiency of the deep learning-based digital volume correlation method of the present disclosure is significantly higher than that of the traditional digital volume correlation method, and the calculation time is reduced by two orders of magnitude.
[0169] The above results confirm the feasibility and beneficial effects of the deep learning-based digital volume correlation method of the embodiment of the present disclosure.
[0170] based on Figure 1 The flowchart of the digital volume correlation method based on deep learning according to an embodiment of the present disclosure is shown. Figure 7 A block diagram of a deep learning-based digital volume correlation device according to an embodiment of the present disclosure is schematically shown.
[0171] like Figure 7As shown, the digital volume correlation device 700 based on deep learning provided by the embodiment of the present disclosure includes a three-dimensional image data set generation module 701, a deep learning network construction module 702, a deep learning network training module 703 and a target displacement field prediction module 704. Among them: the three-dimensional image data set generation module 701 is used to generate a three-dimensional image data set for deep learning network training, the deep learning network construction module 702 is used to build a deep learning network, the deep learning network training module 703 is used to train the deep learning network using the three-dimensional image data set, and the target displacement field prediction module 704 is used to input the three-dimensional digital volume images before and after deformation into the trained deep learning network and perform calculations to obtain a three-dimensional deformation field with sub-voxel accuracy.
[0172] It should be understood that the three-dimensional image data set generation module 701, the deep learning network construction module 702, the deep learning network training module 703 and the target displacement field prediction module 704 can be combined in one module, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.
[0173] According to an embodiment of the present disclosure, at least one of the three-dimensional image data set generation module 701, the deep learning network construction module 702, the deep learning network training module 703, and the target displacement field prediction module 704 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in a proper combination of software, hardware, and firmware. Alternatively, at least one of the three-dimensional image data set generation module 701, the deep learning network construction module 702, the deep learning network training module 703, and the target displacement field prediction module 704 can be at least partially implemented as a computer program module, and when the program is run by a computer, the function of the corresponding module can be executed.
[0174] The disclosed embodiment also provides an electronic device related to digital volume based on deep learning, such as Figure 8 As shown, Figure 8 The electronic device 800 includes: one or more processors 810; a memory 820 storing a computer executable program, which, when executed by the processor 810, enables the processor 810 to implement Figure 1 The deep learning based digital volume correlation method shown.
[0175] Specifically, the processor 810 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 810 may also include an onboard memory for cache purposes. The processor 810 may be a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0176] The memory 820, for example, can be any medium capable of containing, storing, conveying, propagating or transmitting instructions. For example, the readable storage medium can include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device, device or propagation medium. Specific examples of readable storage media include: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a memory, such as a random access memory (RAM) or flash memory; and / or a wired / wireless communication link.
[0177] The memory 820 may include a computer program 821 , which may include code / computer executable instructions, which when executed by the processor 810 causes the processor 810 to perform a method according to an embodiment of the present disclosure or any variation thereof.
[0178] The computer program 821 may be configured to have, for example, a computer program code including a computer program module. For example, in an example embodiment, the code in the computer program 821 may include at least one program module, for example, including a module 821A, a module 821B, ... It should be noted that the division method and number of modules are not fixed, and those skilled in the art may use appropriate program modules or program module combinations according to actual conditions, and when these program module combinations are executed by the processor 810, the processor 810 may execute the method according to the embodiment of the present disclosure or any variation thereof.
[0179] The embodiments of the present disclosure also provide a storage medium containing computer executable instructions, which may be contained in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The storage medium containing computer executable instructions carries one or more programs, and when the one or more programs are executed, the deep learning-based digital volume correlation method according to the embodiments of the present disclosure is implemented.
[0180] According to an embodiment of the present disclosure, the storage medium containing computer executable instructions may be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, optical cable, radio frequency signal, etc., or any suitable combination of the above.
[0181] The present disclosure also provides a computer program, which includes: computer executable instructions, which are used to implement the deep learning-based digital volume correlation method according to an embodiment of the present disclosure when the instructions are executed.
[0182] So far, the present disclosure has been described in detail in conjunction with the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of the present disclosure.
[0183] It should be noted that the implementation methods not shown or described in the drawings or the text of the specification are all forms known to ordinary technicians in the relevant technical field and are not described in detail. In addition, the above definitions of each element are not limited to the various specific structures, shapes or methods mentioned in the embodiments, and ordinary technicians in the field can simply change or replace them.
[0184] Of course, according to actual needs, the present disclosure may also include other parts, which are irrelevant to the innovation of the present disclosure and will not be described here.
[0185] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various disclosed aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed disclosure requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the disclosed aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present disclosure.
[0186] In addition, in the drawings or descriptions, similar or identical parts use the same figure numbers. The technical features in the various embodiments exemplified in the specification can be freely combined to form new solutions without conflict. In addition, each claim can be used as an embodiment alone or the technical features in each claim can be combined as a new embodiment. Furthermore, the elements or implementations not shown or described in the drawings are forms known to ordinary technicians in the relevant technical field. In addition, although demonstrations of parameters containing specific values may be provided herein, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but can be approximated to the corresponding values within an acceptable error tolerance or design constraint.
[0187] Unless there are technical obstacles or contradictions, the above-mentioned various embodiments of the present disclosure can be freely combined to form additional embodiments, and these additional embodiments are all within the protection scope of the present disclosure.
[0188] Although the present disclosure is described in conjunction with the accompanying drawings, the embodiments disclosed in the accompanying drawings are intended to exemplify the preferred embodiments of the present disclosure and should not be construed as limiting the present disclosure. The dimensional ratios in the accompanying drawings are merely illustrative and should not be construed as limiting the present disclosure.
[0189] Although some embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept, the scope of which is defined by the claims and their equivalents.
[0190] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A digital volume correlation method based on deep learning, characterized in that: include: Generate 3D image datasets for deep learning network training; Build deep learning networks; Using 3D image datasets to train deep learning networks; as well as Input the three-dimensional digital volume images before and after deformation into the trained deep learning network and perform calculations to obtain a three-dimensional deformation field with sub-voxel accuracy; Wherein, in the step of constructing a deep learning network, the deep learning network includes the following three networks that perform different functions: a whole voxel correlation network, used to calculate the whole voxel displacement of each sampling point in the three-dimensional digital volume image; a subvoxel correlation network, used to calculate the subvoxel displacement of each sampling point in the three-dimensional digital volume image; a displacement optimization network, used to perform smooth denoising post-processing on the three-dimensional displacement field output by the subvoxel correlation network; The method of inputting the three-dimensional digital volume images before and after deformation into a trained deep learning network and performing calculation to obtain a three-dimensional deformation field with sub-voxel accuracy comprises: the three-dimensional digital volume images before and after deformation to be tested are input into the trained deep learning network; the set of three-dimensional digital volume images to be tested are first mapped by each layer of the whole voxel correlation network to obtain a three-dimensional displacement with whole voxel accuracy and used as a displacement correction of the sub-voxel correlation network; then, the three-dimensional digital volume images before and after deformation are mapped by each layer of the sub-voxel correlation network to obtain a three-dimensional displacement with sub-voxel accuracy; finally, a smooth three-dimensional deformation field is output by a displacement optimization network.
2. The deep learning-based digital volume correlation method according to claim 1, characterized in that: The generating of a three-dimensional image data set for deep learning network training comprises: A series of simulated three-dimensional digital volume images before deformation and simulated three-dimensional digital volume images after deformation are generated as data sets for deep learning network training and verification.
3. The digital volume correlation method based on deep learning according to claim 2, characterized in that: The gray value G0 of any point (X, Y, Z) in the simulated three-dimensional digital volume image before deformation is determined by the following formula: where a x 、a y 、a z are the full width at half maximum of the point spread function along the x, y, and z directions, respectively, (X c,i ,Y c,i ,Z c,i ) is the center position of the i-th speckle particle, b is the background intensity of the three-dimensional simulated image, and n is the number of speckle particles.
4. The deep learning-based digital volume correlation method according to claim 3, characterized in that: The gray value G1 of any point (x, y, z) in the deformed simulated three-dimensional digital volume image is determined by the following formula: Among them, u c,i 、v c,i 、w c,i They are points (X c,i ,Y c,i ,Z c,i ) are the displacement components along the x, y, and z directions.
5. The digital volume correlation method based on deep learning according to claim 1, characterized in that: The whole pixel correlation network comprises a whole pixel image segmentation layer, a first whole pixel three-dimensional convolution layer (11), a first whole pixel three-dimensional maximum pooling layer (21), a second whole pixel three-dimensional convolution layer (12), a second whole pixel three-dimensional maximum pooling layer (22), a third whole pixel three-dimensional convolution layer (13), a third whole pixel three-dimensional maximum pooling layer (23) and a whole pixel fully connected layer. The whole pixel correlation network calculates the whole pixel displacement of each sampling point in the three-dimensional digital volume image, including: First, the input 3D digital volume image before deformation is split into M pixels with sampling points (X s ,Y s ,Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first whole pixel three-dimensional convolution layer (11) together with the corresponding deformed sub-image; wherein the corresponding deformed sub-image is based on the sampling point (x s ,y s ,z s ) as the center, with (D x ,D y ,D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s ,z s ) can be obtained by (X s ,Y s ,Z s ) and displacement correction term (u s ,v s ,w s )Sure: x s =X s +u s and s =And s +v s With s =Z s +in s In the overall voxel correlation network, the displacement correction term is taken as u s =v s =w s =0; Secondly, in the first whole pixel three-dimensional convolution layer (11), a three-dimensional convolution operation is performed on the input three-dimensional sub-image before and after deformation and the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function; Subsequently, the feature map extracted by the convolution operation is subjected to a three-dimensional maximum pooling operation in a first whole-pixel three-dimensional maximum pooling layer (21) to extract main features; Next, the second whole pixel 3D convolution layer (12), the second whole pixel 3D maximum pooling layer (22), the third whole pixel 3D convolution layer (13) and the third whole pixel 3D maximum pooling layer (23) in the whole pixel correlation network further abstract the feature information learned in the previous layer in turn, and finally integrate them through the whole pixel fully connected layer to obtain an output of a specified size (1×1×1×3×M), which is the three displacement components of the whole pixel accuracy of each sampling point.
6. The deep learning-based digital volume correlation method according to claim 1, characterized in that: The subvoxel correlation network comprises a subvoxel image segmentation layer, a first subvoxel three-dimensional convolution layer (31), a first subvoxel three-dimensional maximum pooling layer (41), a second subvoxel three-dimensional convolution layer (32), a second subvoxel three-dimensional maximum pooling layer (42), a third subvoxel three-dimensional convolution layer (33), a third subvoxel three-dimensional maximum pooling layer (43) and a subvoxel fully connected layer. The subvoxel correlation network calculates the subvoxel displacement of each sampling point in the three-dimensional digital volume image, including: First, the input 3D digital image before deformation is split into M subvoxel image segmentation layers with sampling points (X s ,Y s ,Z s ) is the center, and the dimensions in the x, y, and z directions are D x , D y , D z The sub-image is input into the first sub-voxel three-dimensional convolution layer (31) together with the corresponding deformed sub-image; wherein the corresponding deformed sub-image is based on the sampling point (x s ,y s ,z s ) as the center, with (D x ,D y ,D z ) is the local part of the three-dimensional digital image after the size is deformed, and (x s ,y s ,z s ) can be obtained by (X s ,Y s ,Z s ) and displacement correction term (u s ,v s ,w s )Sure: x s =X s +u s and s =And s +v s With s =Z s +in s In the subvoxel correlation network, the displacement correction term is taken as the output of the overall voxel correlation network; Secondly, in the first subvoxel 3D convolution layer (31), the input 3D sub-image before and after deformation is subjected to 3D convolution operation with the convolution kernel to be learned, and nonlinear information is introduced by the ReLU activation function; Subsequently, the feature map extracted by the convolution operation is subjected to a three-dimensional maximum pooling operation in a first sub-voxel three-dimensional maximum pooling layer (41) to extract main features; Next, the second subvoxel 3D convolution layer (32), the second subvoxel 3D maximum pooling layer (42), the third subvoxel 3D convolution layer (33) and the third subvoxel 3D maximum pooling layer (43) in the subvoxel correlation network further abstract the feature information learned in the previous layer in turn, and finally integrate them through the subvoxel fully connected layer to obtain an output of a specified size (1×1×1×3×M), which is the three displacement components of the subvoxel accuracy of each sampling point.
7. The digital volume correlation method based on deep learning according to claim 1, characterized in that: The displacement optimization network includes a local displacement assembly layer, a displacement optimization fully connected layer and a global displacement assembly layer. The displacement optimization network performs smooth denoising post-processing on the three-dimensional displacement field output by the subvoxel correlation network, including: Firstly, the local displacement assembly layer integrates the three-dimensional displacement fields of each input sampling point to obtain the integrated spatial information; Secondly, the displacement-optimized fully connected layer determines the least squares coefficients of the local displacement based on the displacement field after integrating the spatial information; Finally, the global displacement assembly layer outputs the final three-dimensional full-field displacement distribution according to the determined least squares coefficients.
8. The deep learning-based digital volume correlation method according to claim 7, characterized in that: In the step of integrating the input three-dimensional displacement fields of each sampling point by the local displacement assembly layer, the local displacement assembly layer integrates the displacement information of five adjacent sampling points.
9. The deep learning-based digital volume correlation method according to claim 1, characterized in that: The step of training the deep learning network using the three-dimensional image data set is to optimize the parameters to be learned in each layer of the deep learning network using the three-dimensional image data set.
10. The deep learning-based digital volume correlation method according to claim 9, characterized in that: The method of optimizing the parameters to be learned in each layer of the deep learning network by using the three-dimensional image data set specifically includes: 80% of the 3D image data was used to train the deep learning network, 15% of the 3D image data was used to verify the deep learning network, and 5% of the 3D image data was used to test the deep learning network. Among them, the network hyperparameters are determined by random search, the root mean square error is specified as the loss function, and the Adam optimizer is used to optimize the loss. The iteration is terminated when the loss value is less than the specified threshold or the number of iterations is greater than the specified value. The parameter values of each layer of the deep learning network are saved for subsequent prediction of the target displacement field.
11. The deep learning-based digital volume correlation method according to claim 1, characterized in that: The trained deep learning network takes as input the 3D digital volume images before and after deformation and outputs smooth subvoxel accurate full-field deformations.
12. A digital volume correlation device based on deep learning, characterized in that: include: A three-dimensional image data set generation module, used to generate a three-dimensional image data set for deep learning network training; Deep learning network building module, used to build deep learning networks; A deep learning network training module, used to train a deep learning network using a three-dimensional image dataset; as well as The target displacement field prediction module is used to input the three-dimensional digital volume images before and after deformation into the trained deep learning network and perform calculations to obtain the three-dimensional deformation field with sub-voxel accuracy; Wherein, in the step of constructing a deep learning network, the deep learning network includes the following three networks that perform different functions: a whole voxel correlation network, used to calculate the whole voxel displacement of each sampling point in the three-dimensional digital volume image; a subvoxel correlation network, used to calculate the subvoxel displacement of each sampling point in the three-dimensional digital volume image; a displacement optimization network, used to perform smooth denoising post-processing on the three-dimensional displacement field output by the subvoxel correlation network; The method of inputting the three-dimensional digital volume images before and after deformation into a trained deep learning network and performing calculation to obtain a three-dimensional deformation field with sub-voxel accuracy comprises: the three-dimensional digital volume images before and after deformation to be tested are input into the trained deep learning network; the set of three-dimensional digital volume images to be tested are first mapped by each layer of the whole voxel correlation network to obtain a three-dimensional displacement with whole voxel accuracy and used as a displacement correction of the sub-voxel correlation network; then, the three-dimensional digital volume images before and after deformation are mapped by each layer of the sub-voxel correlation network to obtain a three-dimensional displacement with sub-voxel accuracy; finally, a smooth three-dimensional deformation field is output by a displacement optimization network.
13. An electronic device comprising: processor; A memory storing a computer executable program, which, when executed by the processor, enables the processor to execute the deep learning-based digital volume correlation method described in any one of claims 1 to 11.
14. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the deep learning-based digital volume correlation method described in any one of claims 1 to 11 is implemented.
15. A computer program product, characterized in that include: Computer executable instructions, which, when executed by a processor, are used to implement the deep learning-based digital volume correlation method described in any one of claims 1-11.
