Three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of three-dimensional reconstruction. The method comprises the steps that multiple frames of ultrasonic images of an ultrasonic video obtained by scanning an organ to be subjected to three-dimensional reconstruction through an ultrasonic probe are obtained, all the ultrasonic images are input into a time sequence feature extraction network to obtain sequence-level features, the sequence-level features comprise time sequence features with the number the same as that of all the ultrasonic images, and the sequence-level features are input into a pose estimation network to estimate the pose of the organ to be subjected to three-dimensional reconstruction. And calculating pose information of each ultrasonic image, and performing three-dimensional reconstruction on each ultrasonic image according to each pose information to obtain a three-dimensional model of the organ to be subjected to three-dimensional reconstruction. An ultrasonic video is processed through a time sequence feature extraction network and a pose estimation network, so that high-quality three-dimensional reconstruction is completed under the condition that a precise mechanical device is not used.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, device, electronic equipment and storage medium. Background Art

[0002] Ultrasound imaging, as a non-invasive, real-time medical imaging method, has been widely used in clinical diagnosis and interventional treatment. Traditional 3D ultrasound reconstruction technology relies primarily on voxel reconstruction using a dedicated robotic arm. This technique uses precision mechanical devices to control the probe's trajectory and ensure accurate spatial position information. However, this method is expensive and complex to use, making it difficult to implement in routine clinical settings.

[0003] However, manually scanning ultrasound images and performing three-dimensional reconstruction using image registration and fusion faces the problems of fixed scanning techniques, long scanning time, cumbersome steps, low registration accuracy, and large reconstruction errors. Summary of the Invention

[0004] In view of this, an object of embodiments of the present invention is to provide a 3D reconstruction method, apparatus, electronic device, and storage medium to at least partially improve the above-mentioned problem.

[0005] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, an embodiment of the present invention provides a three-dimensional reconstruction method, comprising: Acquire an ultrasound video; the ultrasound video is obtained by scanning the organ to be three-dimensionally reconstructed with an ultrasound probe, and the ultrasound video includes multiple frames of ultrasound images; Inputting each of the ultrasound images into a temporal feature extraction network to obtain sequence-level features; the sequence-level features include the same number of temporal features as the number of each of the ultrasound images; Inputting the sequence-level features into a pose estimation network to calculate the pose information of each of the ultrasound images; According to the posture information, each of the ultrasound images is three-dimensionally reconstructed to obtain a three-dimensional model of the organ to be three-dimensionally reconstructed.

[0006] Optionally, the pose estimation network includes a local correlation calculation module and an affine transformation calculation module, and inputting the sequence-level features into the pose estimation network to calculate the pose information of each of the ultrasound images includes: Calculating a local correlation graph between any two consecutive time series features through the local correlation calculation module; wherein each position of the local correlation graph represents the similarity between the two time series features at the corresponding position; Inputting each of the local correlation maps into the affine transformation calculation module to calculate an affine transformation matrix between the two frames of ultrasound images corresponding to each of the local correlation maps; wherein the affine transformation matrix represents the spatial offset between the two frames of ultrasound images; The first frame of the ultrasound image is taken as the starting point of the world coordinate system, and its posture information is set as the unit matrix. The posture information of the remaining ultrasound images is calculated according to the affine transformation matrix; wherein the posture information of the remaining ultrasound images is obtained by multiplying the posture information of the ultrasound image of the previous frame by its own affine transformation matrix.

[0007] Optionally, calculating the local correlation graph between any two consecutive time series features by the local correlation calculation module includes: For any two consecutive time series features, take a feature block of a preset window size with any position of the two time series features as the center; Calculating a correlation score between the two feature blocks; Each position of the two time series features is traversed, and a correlation score of each position of the two time series features is calculated to obtain a local correlation graph between any two consecutive time series features.

[0008] Optionally, the affine transformation calculation module includes at least one convolution layer, a pooling layer, and a regression head, and inputting each of the local correlation maps into the affine transformation calculation module to calculate the affine transformation matrix between the two frames of the ultrasound images corresponding to each of the local correlation maps includes: For each of the local correlation maps, inputting the local correlation maps into the convolutional layers in sequence to obtain a feature map of the local correlation map; Inputting the feature map into the pooling layer to obtain a feature vector of the local correlation map; Input the feature vector into the regression head to obtain 6 affine transformation parameters; The affine transformation parameters are combined into an affine transformation matrix.

[0009] Optionally, the temporal feature extraction network includes a convolutional neural network and a temporal encoder, and inputting each of the ultrasound images into the temporal feature extraction network to obtain sequence-level features includes: Inputting each of the ultrasound images into the convolutional neural network to obtain spatial features of each of the ultrasound images; stacking the spatial features in sequence according to the order of the ultrasound videos to obtain stacked spatial features; The stacked spatial features are input into the temporal encoder to obtain sequence-level features; the sequence-level features include multiple temporal features with contextual structures and temporal relationships.

[0010] Optionally, the method further includes an optimization step of the temporal feature extraction network and the pose estimation network, the step comprising: For any two consecutive frames of the ultrasound image, calculating the image reconstruction loss and the smoothing loss of the two frames of the ultrasound image according to the two frames of the ultrasound image and their corresponding affine transformation matrices; For any three consecutive frames of the ultrasonic image, input the temporal features of the first frame of the ultrasonic image and the third frame of the ultrasonic image into the local correlation calculation module and the affine transformation calculation module to obtain the interval affine transformation matrix of the first frame of the ultrasonic image and the third frame of the ultrasonic image; Calculating loop consistency losses of the three consecutive frames of ultrasound images according to two affine transformation matrices and an interval affine transformation matrix of the three consecutive frames of ultrasound images; Performing weighted summation on the image reconstruction losses, the smoothing losses, and the loop consistency losses to obtain total loss information; Parameters of the temporal feature extraction network and the pose estimation network are optimized according to the total loss information.

[0011] Optionally, the image reconstruction loss is:

[0012]

[0013] in, represents the first real image of any two consecutive frames of ultrasound images, represents the affine transformation matrix of any two consecutive frames of the ultrasound image, represents the image of the second frame of the real image of any two consecutive frames of the ultrasound image after affine transformation, is the second real image of any two consecutive frames of ultrasound images, for and The structural similarity of is the weight coefficient; The smoothing loss is:

[0014] in, is the position information of the ultrasound image of the t-th frame; The loop consistency loss is:

[0015] in, is the interval affine transformation matrix, 、 are the two affine transformation matrices.

[0016] In a second aspect, an embodiment of the present invention provides a three-dimensional reconstruction device, comprising: An ultrasound video acquisition unit, configured to acquire an ultrasound video; the ultrasound video is acquired by scanning the organ to be 3D reconstructed with an ultrasound probe, and the ultrasound video includes multiple frames of ultrasound images; a sequence-level feature extraction unit, configured to input each of the ultrasound images into a temporal feature extraction network to obtain sequence-level features; the sequence-level features including the same number of temporal features as the number of each of the ultrasound images; a pose information estimation unit, configured to input the sequence-level features into a pose estimation network to calculate the pose information of each of the ultrasound images; The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction on each of the ultrasound images according to the posture information to obtain a three-dimensional model of the organ to be three-dimensionally reconstructed.

[0017] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0018] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements any of the above-described methods when executed by a processor.

[0019] The present invention provides a 3D reconstruction method, device, electronic device, and storage medium. These methods extract temporal features from ultrasound images and, using a pose estimation network, calculate the pose information of each ultrasound image for 3D reconstruction. This enables high-quality 3D reconstruction of ultrasound images without requiring additional equipment or restricting the physician's scanning technique.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention; Figure 2 A schematic flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention; Figure 3 Another schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention; Figure 4 Another schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention; Figure 5 A schematic diagram of a process of step S231 provided in an embodiment of the present invention; Figure 6 A schematic diagram of a process flow of step S232 provided in an embodiment of the present invention; Figure 7 A schematic diagram of a flow chart of the steps for optimizing a temporal feature extraction network and a pose estimation network provided by an embodiment of the present invention; Figure 8 A schematic diagram of the execution process of a three-dimensional reconstruction method provided by an embodiment of the present invention; Figure 9 A schematic structural block diagram of a three-dimensional reconstruction device provided by an embodiment of the present invention.

[0023] Icon: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - communication bus; 400 - three-dimensional reconstruction device; 410 - ultrasound video acquisition unit; 420 - sequence-level feature extraction unit; 430 - posture information estimation unit; 440 - three-dimensional reconstruction unit. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are used only to distinguish the description and should not be understood as indicating or implying relative importance.

[0027] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0028] Traditional 3D ultrasound reconstruction technology relies primarily on voxel reconstruction using a dedicated robotic arm. This technique uses precision mechanical devices to control the probe's trajectory and ensure accurate spatial position information. However, this method is expensive and complex to use, making it difficult to implement in routine clinical settings. Manually scanning ultrasound images and performing 3D reconstruction using image registration and fusion also presents challenges such as fixed scanning techniques, long scanning times, cumbersome procedures, low registration accuracy, and large reconstruction errors.

[0029] Deep learning has been widely used in ultrasound image processing, such as image enhancement, registration, and segmentation. However, for 3D modeling, most methods still rely on manually defined motion trajectories or external equipment, limiting the algorithm's generalization and clinical application.

[0030] Based on the above, embodiments of the present invention provide a 3D reconstruction method, apparatus, electronic device, and storage medium. Through the coordinated processing of a temporal feature extraction network and a pose estimation network, the pose information of each ultrasound image is calculated and 3D reconstruction is performed based on this pose information. This enables high-quality 3D reconstruction of ultrasound images without requiring additional equipment or restricting the physician's scanning technique.

[0031] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103. The memory 101 and processor 103 are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101, thereby performing various functional applications and data processing.

[0032] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, or the like. It is understood that the electronic device 100 is not limited to a physical server and may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, or other computer that provides the same functionality as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, Windows, Linux, or the like.

[0033] The memory 101 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0034] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0035] Processor 103 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the embodiments of the present invention may be completed by hardware integrated logic circuits in processor 103 or by software instructions. Processor 103 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0036] I understand. Figure 1 The structure shown is for illustration only. The electronic device 100 may further include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0037] The following is an exemplary description of the three-dimensional reconstruction method provided by the present invention. Figure 2 A schematic diagram of a three-dimensional reconstruction method according to an embodiment of the present invention is provided. Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes the following steps: Figure 2 The following steps are described: S210: Acquire an ultrasound video; wherein the ultrasound video is obtained by scanning the organ to be three-dimensionally reconstructed with an ultrasound probe, and the ultrasound video includes multiple frames of ultrasound images.

[0038] S220: Input each ultrasound image into a temporal feature extraction network to obtain sequence-level features; wherein the sequence-level features include the same number of temporal features as the number of ultrasound images.

[0039] S230: Inputting the sequence-level features into the pose estimation network to calculate the pose information of each ultrasound image.

[0040] S240: Perform three-dimensional reconstruction on each ultrasound image according to each pose information to obtain a three-dimensional model of the organ to be three-dimensionally reconstructed.

[0041] To perform three-dimensional reconstruction of an organ to be three-dimensionally reconstructed using the three-dimensional reconstruction method, it is necessary to first obtain an ultrasound video containing multiple frames of ultrasound images. The ultrasound video can be obtained by scanning the organ to be three-dimensionally reconstructed with an ultrasound probe.

[0042] Next, each ultrasound image frame in the ultrasound video is fed into a temporal feature extraction network, which extracts a number of sequence-level features equal to the number of ultrasound images. These sequence-level features include the temporal features corresponding to each ultrasound frame, reflecting the temporal relationship between images and the underlying characteristic patterns.

[0043] Then, the extracted sequence-level features are fed into the pose estimation network to calculate the pose information corresponding to each frame of ultrasound image. The pose information is used to describe the spatial position and direction of the ultrasound probe when acquiring the frame of ultrasound image.

[0044] Finally, using the obtained pose information for each ultrasound frame, all ultrasound images are spatially aligned and fused to complete the 3D model of the organ to be reconstructed. This process generates intuitive and accurate 3D models, providing important support for medical diagnosis and research.

[0045] This method extracts temporal features from ultrasound images and uses a pose estimation network to calculate the pose information of each ultrasound image for 3D reconstruction. This allows for high-quality 3D reconstruction of ultrasound images without requiring additional equipment or restricting the doctor's scanning technique.

[0046] After acquiring ultrasound video, to improve 3D reconstruction, preprocessing can be performed. Each frame is first denoised and normalized, and images with insufficient clarity are removed. A quality assessment algorithm can be used for automated screening, calculating the variance of the gradient amplitude for each frame and removing images with variances less than a preset value.

[0047] The specific method for performing 3D reconstruction in step S240 can be to use the first frame of the ultrasound image as the origin of the world coordinate system, map all ultrasound images to the world coordinate system based on their pose information, and complete 3D voxel fusion using the Truncated Signed Distance Function (TSDF). The TSDF can effectively handle noise and uncertainty between different frames. By averaging and fusing multi-frame observations, it improves the consistency and density of voxel data and is robust to small pose errors during the scanning process. Finally, a voxel-level reconstruction method (MarchingCubes, MC) is used to extract a 3D surface mesh model to obtain a 3D model of the organ to be 3D reconstructed. MC can efficiently and accurately extract continuous, closed isosurfaces from dense voxels, making it ideal for medical organ surface modeling. The generated mesh can be directly used for subsequent clinical analysis or 3D printing.

[0048] In order to extract the spatial features of each ultrasound image frame in the ultrasound video and the temporal contextual relationship between each ultrasound image frame, in a possible implementation, the temporal feature extraction network may include a convolutional neural network and a temporal encoder, see Figure 3 , the above step S220 may include the following steps: S221: Input each ultrasound image into a convolutional neural network to obtain spatial features of each ultrasound image.

[0049] Among them, the convolutional neural network can be a convolutional neural network based on ResNet.

[0050] S222: stacking the spatial features in sequence according to the order of the ultrasound video to obtain a stacked spatial feature.

[0051] S223: Input the stacked spatial features into the temporal encoder to obtain sequence-level features; wherein the sequence-level features include multiple temporal features with contextual structures and temporal relationships.

[0052] The temporal encoder may be a Transformer encoder.

[0053] A convolutional neural network is used to extract the spatial features of each frame of ultrasound image. For example, an ultrasound video includes T ultrasound images, and the size of each ultrasound image is , then the input of the convolutional neural network is , after convolutional neural network encoding, the spatial features of each frame of ultrasound image are , according to the order of ultrasound video, these spatial features are stacked in sequence to form a four-dimensional tensor , C represents the number of feature dimensions of each frame of ultrasound image after encoding. This four-dimensional tensor is input into the temporal encoder, and the temporal context of each ultrasound image is captured through the global self-attention mechanism. The output sequence-level features are: .

[0054] In order to more effectively capture the fine-grained spatial transformation relationship between frames, a local correlation regression method can be designed to perform relative pose estimation to improve the sensitivity to small structural changes in ultrasound image sequences and the accuracy of pose recovery. In one possible implementation, the pose estimation network can include a local correlation calculation module and an affine transformation calculation module, see Figure 4 , the above step S230 may include the following steps: S231: Calculate a local correlation graph between any two consecutive time series features through a local correlation calculation module; wherein each position of the local correlation graph represents the similarity between the two time series features at the corresponding position.

[0055] After obtaining the sequence-level features, we first calculate the local correlation graph between any two consecutive time series features, for example , which is a 10×10 feature map. The similarity of each point of the two time series features is calculated. The similarity can directly calculate the dot product or cosine similarity of the two points to obtain a 10×10 local correlation map. Each point of the local correlation map represents a correlation score.

[0056] In order to capture more detailed local information between each time series feature, two consecutive time series features can be Calculate the local correlation graph of the two-frame time series features in a local sliding window manner. Optionally, see Figure 5 , step S231 may include the following steps: S2311: For any two consecutive time series features, take a feature block of a preset window size with any position of the two time series features as the center.

[0057] S2312: Calculate the correlation scores of the two feature blocks.

[0058] S2313: Traverse each position of the two time series features, calculate the correlation score of each position of the two time series features, and obtain a local correlation graph between any two consecutive time series features.

[0059] For any two consecutive time series features , in the timing characteristics In each position Center and take the preset window size Local feature blocks , also in the timing characteristics Take the feature block at the corresponding position , and then calculate the local correlation score between these two feature blocks , the correlation score can be calculated by using methods such as dot product and cosine similarity. Finally, the two time series features are traversed All positions of , a local correlation graph is obtained.

[0060] Among them, the formula for calculating the correlation score using dot product is:

[0061] The formula for calculating the relevance score using cosine similarity is:

[0062] Among them, u and v are the feature points inside the feature block relative to the center point The offset of .

[0063] It is understandable that in practice, the sizes of feature blocks are not completely consistent. For example, the preset window size is 3×3. In the feature map, the feature blocks at the four corners only include 4 feature points, the feature blocks on the four sides include 6 feature points, and the feature blocks in other places include 9 feature points.

[0064] S232: Inputting each local correlation map into an affine transformation calculation module to calculate an affine transformation matrix between two frames of ultrasound images corresponding to each local correlation map; wherein the affine transformation matrix represents the spatial offset between the two frames of ultrasound images.

[0065] The calculated local correlation map is used as input and processed by the affine transformation calculation module to find the affine transformation matrix of the correspondence between the two frames of ultrasound images. There are many ways to process the local correlation map. In one possible implementation, the affine transformation calculation module may include at least one convolution layer, a pooling layer, and a regression head. Figure 6 , S232 may include the following steps: S2321: For each local correlation map, the local correlation map is input into each convolution layer in sequence to obtain a feature map of the local correlation map.

[0066] S2322: Input the feature map into the pooling layer to obtain a feature vector of the local correlation map.

[0067] S2323: Input the feature vector into the regression head to obtain 6 affine transformation parameters.

[0068] S2324: Combine the affine transformation parameters into an affine transformation matrix.

[0069] Exemplarily, the affine transformation calculation module can be a lightweight convolutional neural network, which consists of three 3×3 convolutional layers with 32, 64, and 128 channels, respectively. Each convolution layer is then activated with BatchNorm and ReLU. The output of the convolution layer is globally averaged and pooled using a pooling layer to compress the convolution output into a fixed-dimensional (e.g., 128-dimensional) feature vector. Finally, the feature vector is passed through a regression head to output the affine transformation parameters between the two frames. The regression head can be composed of two fully connected layers, and the last fully connected layer has an output channel of 6, that is, it outputs 6 affine transformation parameters, including 3 rotation parameters and 3 translation parameters: .

[0070] These six affine transformation parameters are then combined into a standard three-dimensional affine transformation matrix. A standard three-dimensional affine transformation matrix is a 4×4 matrix. The upper left column shows a 3×3 matrix converted from the three rotation parameters. The first three rows in the fourth column are the translation vectors, i.e., the three translation parameters, and the last row is the homogeneous coordinates.

[0071] S233: Take the first frame of ultrasound image as the starting point of the world coordinate system, and set its posture information as the unit matrix, and calculate the posture information of the remaining ultrasound images according to the affine transformation matrix; wherein the posture information of the remaining ultrasound images is obtained by multiplying its own affine transformation matrix by the posture information of the previous frame of ultrasound image.

[0072] After obtaining the affine transformation matrix between any two consecutive ultrasound image frames, in order to perform 3D reconstruction faster, the pose information of each ultrasound image frame can be calculated. First, the first ultrasound image frame is used as the starting point of the world coordinate system, and its pose information is set to the unit matrix. Then the pose information of the first ultrasound image frame is The remaining posture information of each ultrasound image is accumulated through continuous transformation , is the affine transformation matrix between the ultrasound images of the t-1th frame and the tth frame.

[0073] In one possible implementation, the method may further include steps for optimizing the temporal feature extraction network and the pose estimation network, see Figure 7 , which includes: S310: For any two consecutive ultrasonic image frames, calculate the image reconstruction loss and smoothing loss of the two ultrasonic image frames according to the two ultrasonic image frames and their corresponding affine transformation matrices.

[0074] In the above process, the affine transformation matrix of any two consecutive ultrasonic image frames has been obtained. Therefore, the image reconstruction loss and smoothing loss of the two ultrasonic image frames can be directly calculated based on the two ultrasonic image frames and their corresponding affine transformation matrices.

[0075] Optionally, the image reconstruction loss can be:

[0076]

[0077] in, represents the first real image of any two consecutive ultrasound images, Represents the affine transformation matrix of any two consecutive ultrasound image frames, It represents the image of the second frame of any two consecutive ultrasound images after affine transformation. is the second real image of any two consecutive ultrasound images, for and The structural similarity of is the weight coefficient.

[0078] Optionally, the smoothing loss can be:

[0079] in, is the pose information of the t-th frame ultrasound image.

[0080] S320: For any three consecutive ultrasonic image frames, the temporal features of the first ultrasonic image frame and the third ultrasonic image frame are input into the local correlation calculation module and the affine transformation calculation module to obtain the interval affine transformation matrix of the first ultrasonic image frame and the third ultrasonic image frame.

[0081] To ensure that the transformation of any three consecutive ultrasound frames satisfies the geometric loop relationship, a loss function needs to be established for the three consecutive ultrasound frames. Based on the affine transformation matrix of any two consecutive ultrasound frames, the temporal features of the first and third ultrasound frames of the three consecutive ultrasound frames are input into the local correlation calculation module and the affine transformation calculation module. The affine transformation matrix of the first and third ultrasound frames is recorded as the interval affine transformation matrix.

[0082] S330: Calculating the loop consistency loss of the three frames of ultrasound images according to two affine transformation matrices and an interval affine transformation matrix of the three consecutive frames of ultrasound images.

[0083] After obtaining the affine transformation matrices of any two frames of ultrasound images in three consecutive frames of ultrasound images, the loop consistency loss of the three frames of ultrasound images can be calculated based on these affine transformation matrices.

[0084] Optionally, the loop consistency loss can be:

[0085] in, is the interval affine transformation matrix, 、 are two affine transformation matrices.

[0086] S340: Perform weighted summation on each image reconstruction loss, each smoothing loss, and each loop consistency loss to obtain total loss information.

[0087] The total loss information can be recorded as:

[0088] in, is the weight coefficient of each loss.

[0089] S350: Optimize parameters of the temporal feature extraction network and the pose estimation network based on the total loss information.

[0090] Finally, based on the total loss information, we can optimize the parameters of the time series feature extraction network and the pose estimation network. This optimization method does not require annotation and can further optimize the network during use.

[0091] It is understandable that before using this 3D reconstruction method, the temporal feature extraction network and the pose estimation network can also be pre-trained. First, an ultrasound video sample is obtained and processed using the above 3D reconstruction method to obtain the affine transformation matrix of any two needle images in any three consecutive ultrasound frames. The total loss information is calculated using the above image reconstruction loss, smoothing loss, and loop consistency loss calculation methods, and the parameters of the temporal feature extraction network and the pose estimation network are updated based on the total loss information.

[0092] In an alternative embodiment, see Figure 8 First, an ultrasound scan is performed on the organ to be 3D reconstructed to obtain an ultrasound video. The ultrasound video is then denoised, normalized, and key frames are filtered to obtain the processed ultrasound video. The ultrasound video is input into CNN and Transformer for processing to obtain the temporal features of each ultrasound image. Based on each temporal feature, the local correlation map of two consecutive temporal features is calculated. The local correlation map is input into a lightweight CNN to obtain the affine transformation parameters of the two ultrasound images, which are further converted into an affine transformation matrix. Based on each affine transformation matrix, the posture information of each ultrasound image is calculated. Based on the posture information and the ultrasound image, the 3D voxel fusion is completed through TSDF. Finally, MC is used to extract the 3D surface mesh model to obtain the 3D model of the organ to be 3D reconstructed.

[0093] Furthermore, the embodiment of the present invention also provides a three-dimensional reconstruction device, see Figure 9 , the three-dimensional reconstruction device 400 includes: The ultrasound video acquisition unit 410 is used to acquire ultrasound video. The ultrasound video is acquired by scanning the organ to be 3D reconstructed with an ultrasound probe. The ultrasound video includes multiple frames of ultrasound images.

[0094] The sequence-level feature extraction unit 420 is used to input each ultrasound image into a temporal feature extraction network to obtain sequence-level features; the sequence-level features include the same number of temporal features as the number of ultrasound images.

[0095] The pose information estimation unit 430 is used to input the sequence-level features into the pose estimation network to calculate the pose information of each ultrasound image.

[0096] The three-dimensional reconstruction unit 440 is used to perform three-dimensional reconstruction on each ultrasound image according to each position information to obtain a three-dimensional model of the organ to be three-dimensionally reconstructed.

[0097] In summary, the embodiments of the present invention provide a 3D reconstruction method, apparatus, electronic device, and storage medium. These methods extract temporal features from ultrasound images and, through a pose estimation network, calculate the pose information of each ultrasound image to perform 3D reconstruction. This allows for high-quality 3D reconstruction of ultrasound images without the need for additional equipment or restrictions on the physician's scanning technique. By introducing temporal consistency constraints, the method automatically adapts to different scanning rates and trajectories, improving the robustness and accuracy of reconstruction. Based on a fully self-supervised training framework, the model can be iteratively optimized on large-scale clinical data.

[0098] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0100] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0101] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0102] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire an ultrasound video; the ultrasound video is obtained by scanning the organ to be three-dimensionally reconstructed with an ultrasound probe, and the ultrasound video includes multiple frames of ultrasound images; Inputting each of the ultrasound images into a temporal feature extraction network to obtain sequence-level features; the sequence-level features include the same number of temporal features as the number of each of the ultrasound images; Inputting the sequence-level features into a pose estimation network to calculate the pose information of each of the ultrasound images; According to the posture information, each of the ultrasound images is three-dimensionally reconstructed to obtain a three-dimensional model of the organ to be three-dimensionally reconstructed.

2. The method according to claim 1, characterized in that The pose estimation network includes a local correlation calculation module and an affine transformation calculation module. The sequence-level features are input into the pose estimation network to calculate the pose information of each ultrasound image, including: Calculating a local correlation graph between any two consecutive time series features through the local correlation calculation module; wherein each position of the local correlation graph represents the similarity between the two time series features at the corresponding position; Inputting each of the local correlation maps into the affine transformation calculation module to calculate an affine transformation matrix between the two frames of ultrasound images corresponding to each of the local correlation maps; wherein the affine transformation matrix represents the spatial offset between the two frames of ultrasound images; The first frame of the ultrasound image is taken as the starting point of the world coordinate system, and its posture information is set as the unit matrix. The posture information of the remaining ultrasound images is calculated according to the affine transformation matrix; wherein the posture information of the remaining ultrasound images is obtained by multiplying the posture information of the ultrasound image of the previous frame by its own affine transformation matrix.

3. The method according to claim 2, characterized in that The calculating of the local correlation graph between any two consecutive time series features by the local correlation calculation module includes: For any two consecutive time series features, take a feature block of a preset window size with any position of the two time series features as the center; Calculating a correlation score between the two feature blocks; Each position of the two time series features is traversed, and a correlation score of each position of the two time series features is calculated to obtain a local correlation graph between any two consecutive time series features.

4. The method according to claim 2, characterized in that The affine transformation calculation module includes at least one convolution layer, a pooling layer, and a regression head. Inputting each local correlation map into the affine transformation calculation module and calculating the affine transformation matrix between the two frames of ultrasound images corresponding to each local correlation map include: For each of the local correlation maps, inputting the local correlation maps into the convolutional layers in sequence to obtain a feature map of the local correlation map; Inputting the feature map into the pooling layer to obtain a feature vector of the local correlation map; Input the feature vector into the regression head to obtain 6 affine transformation parameters; The affine transformation parameters are combined into an affine transformation matrix.

5. The method according to claim 1, wherein The temporal feature extraction network includes a convolutional neural network and a temporal encoder. Inputting each of the ultrasound images into the temporal feature extraction network to obtain sequence-level features includes: Inputting each of the ultrasound images into the convolutional neural network to obtain spatial features of each of the ultrasound images; stacking the spatial features in sequence according to the order of the ultrasound videos to obtain stacked spatial features; The stacked spatial features are input into the temporal encoder to obtain sequence-level features; the sequence-level features include multiple temporal features with contextual structures and temporal relationships.

6. The method according to claim 2, characterized in that The method further includes an optimization step for the temporal feature extraction network and the pose estimation network, the step comprising: For any two consecutive frames of the ultrasound image, calculate the image reconstruction loss and smoothing loss of the two frames of the ultrasound image according to the two frames of the ultrasound image and their corresponding affine transformation matrices; For any three consecutive frames of the ultrasonic image, input the temporal features of the first frame of the ultrasonic image and the third frame of the ultrasonic image into the local correlation calculation module and the affine transformation calculation module to obtain the interval affine transformation matrix of the first frame of the ultrasonic image and the third frame of the ultrasonic image; Calculating loop consistency losses of the three consecutive frames of ultrasound images according to two affine transformation matrices and an interval affine transformation matrix of the three consecutive frames of ultrasound images; Performing weighted summation on the image reconstruction losses, the smoothing losses, and the loop consistency losses to obtain total loss information; Parameters of the temporal feature extraction network and the pose estimation network are optimized according to the total loss information.

7. The method according to claim 6, characterized in that The image reconstruction loss is: in, represents the first real image of any two consecutive frames of ultrasound images, represents the affine transformation matrix of any two consecutive frames of the ultrasound image, represents the image of the second frame of the real image of any two consecutive frames of the ultrasound image after affine transformation, is the second real image of any two consecutive frames of ultrasound images, for and The structural similarity of is the weight coefficient; The smoothing loss is: in, is the position information of the ultrasound image of the t-th frame; The loop consistency loss is: in, is the interval affine transformation matrix, 、 are the two affine transformation matrices.

8. A three-dimensional reconstruction device, characterized in that: include: An ultrasound video acquisition unit, configured to acquire an ultrasound video; the ultrasound video is acquired by scanning the organ to be 3D reconstructed with an ultrasound probe, and the ultrasound video includes multiple frames of ultrasound images; a sequence-level feature extraction unit, configured to input each of the ultrasound images into a temporal feature extraction network to obtain sequence-level features; the sequence-level features including the same number of temporal features as the number of each of the ultrasound images; a pose information estimation unit, configured to input the sequence-level features into a pose estimation network to calculate the pose information of each of the ultrasound images; The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction on each of the ultrasound images according to the posture information to obtain a three-dimensional model of the organ to be three-dimensionally reconstructed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images