A three-dimensional reconstruction method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a three-dimensional reconstruction method, device, electronic device and storage medium, which relates to the field of image processing technology. By acquiring all ultrasound images of the target gland, and calculating the first coordinate information of each ultrasound image in the world coordinate system, each ultrasound image is input into the gland segmentation network to obtain a gland contour mask map, each ultrasound image is input into the nodule detection network to obtain a nodule contour mask map, and each nodule contour mask map is mapped to each gland contour mask map, and the gland mask map corresponding to each ultrasound image is obtained, any gland mask map is selected as the first gland mask map, and the remaining gland mask maps are used as the second gland mask map, and the relative pixel distance of each second gland mask map relative to the first gland mask map is calculated according to the first coordinate information, and the pixel coordinate data of each gland mask map is placed in a three-dimensional matrix to obtain a gland three-dimensional matrix. Thus, a three-dimensional model of the gland can be quickly constructed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional reconstruction method, device, electronic equipment and storage medium. Background Art

[0002] In medical imaging, existing 3D reconstruction is usually based on data obtained from computer tomography (CT) or magnetic resonance imaging (MRI). Through 3D reconstruction, the location of thyroid nodules or tumors can be located, their size and relationship with surrounding structures can be understood, and a more accurate plan can be provided for subsequent treatment.

[0003] The existing 3D reconstruction of the thyroid gland requires patients to undergo CT or MRI examinations, and the entire thyroid gland is reconstructed through the data obtained from the tomography. However, for patients, undergoing additional CT or MRI examinations requires more time and money, and also has a certain impact on the patient's body. Summary of the invention

[0004] In view of this, an object of embodiments of the present invention is to provide a 3D reconstruction method, device, electronic device and storage medium to at least partially improve the above-mentioned problem.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a three-dimensional reconstruction method, which is applied to an electronic device of an ultrasonic inspection system, wherein the ultrasonic inspection system further includes an ultrasonic probe, and the method includes:

[0007] Acquire all ultrasonic images of the target gland, and calculate the first coordinate information of each ultrasonic image in the world coordinate system; the ultrasonic image is obtained by detection by the ultrasonic probe;

[0008] Inputting each of the ultrasound images into a gland segmentation network to obtain a gland contour mask map, inputting each of the ultrasound images into a nodule detection network to obtain a nodule contour mask map, and mapping each of the nodule contour mask maps to each of the gland contour mask maps to obtain a gland mask map corresponding to each of the ultrasound images;

[0009] Selecting any one of the gland mask images as a first gland mask image, and taking the remaining gland mask images as second gland mask images, and calculating a relative pixel distance of each of the second gland mask images relative to the first gland mask image according to the first coordinate information;

[0010] A three-dimensional matrix is ​​created, and pixel coordinate data of each gland mask image is placed into the three-dimensional matrix according to each relative pixel distance, so as to obtain a gland three-dimensional matrix.

[0011] Optionally, the ultrasonic inspection system further includes a sensor, which is disposed on the ultrasonic probe and offset by a preset distance in one direction relative to the vertex of the ultrasonic probe. The step of calculating the first coordinate information of each of the ultrasonic images in the world coordinate system includes:

[0012] Constructing a local coordinate system with the center of the sensor as the origin, and determining second coordinate information of the vertex in the local coordinate system according to the positions of the center and the vertex;

[0013] Based on the first coordinate information calculation formula, the first coordinate information of the ultrasonic image in the world coordinate system is calculated according to the third coordinate information of the sensor in the world coordinate system and the second coordinate information.

[0014] Optionally, the first coordinate information calculation formula is:

[0015] YPR*(a,b,c)+(x,y,z)

[0016] Among them, (a, b, c) is the second coordinate information, (x, y, z) is the coordinate of the third coordinate information, and Y, P, R is the rotation matrix corresponding to the Euler angle of the third coordinate information.

[0017] Optionally, the step of mapping each of the nodule contour mask images to each of the gland contour mask images to obtain a gland mask image corresponding to each of the ultrasound images comprises:

[0018] When the nodule contour mask image is not empty, calculating the superposition value of each pixel in the nodule contour mask image according to a preset weight value;

[0019] The calculated superposition values ​​are added to corresponding pixels of the gland contour mask image to obtain the gland mask image.

[0020] Optionally, the method further comprises:

[0021] Detecting whether there is nodule data in the gland mask image; a gland mask image includes at least zero nodule data;

[0022] If yes, based on the gland mask image, each nodule data is marked with a different color according to each nodule data.

[0023] Optionally, the step of selecting any one of the gland mask images as the first gland mask image, using the remaining gland mask images as the second gland mask images, and calculating the relative pixel distance of each of the second gland mask images relative to the first gland mask image according to the first coordinate information includes:

[0024] subtracting the first coordinate information of the first ultrasonic image from the first coordinate information of each second ultrasonic image to obtain the relative physical coordinates of each second ultrasonic image;

[0025] Each of the relative physical coordinates is converted into a relative pixel distance according to a pixel distance conversion coefficient of the ultrasonic image.

[0026] Optionally, the step of creating a three-dimensional matrix includes:

[0027] Determining the dimension of the three-dimensional matrix according to the pixels of the ultrasound image and the relative pixel distance;

[0028] Initialize the three-dimensional matrix according to the dimension.

[0029] Optionally, the step of placing pixel coordinate data of each gland mask image into the three-dimensional matrix according to each relative pixel distance to obtain the gland three-dimensional matrix comprises:

[0030] According to pixel coordinates of the first gland mask image, placing the first gland mask image into the three-dimensional matrix;

[0031] According to the pixel coordinates of the second gland mask image and the relative pixel distance, the second gland mask image is placed in the three-dimensional matrix to obtain the gland three-dimensional matrix.

[0032] Optionally, the method further comprises:

[0033] Calculating the gradient and gradient amplitude of the three-dimensional matrix of the gland;

[0034] According to the preset diffusion function, the diffusion coefficient of each voxel of the gland three-dimensional matrix is ​​calculated; the diffusion coefficient is:

[0035]

[0036] Wherein, c is the diffusion coefficient, g is the diffusion function, is the gradient amplitude, k is a positive constant;

[0037] Each voxel is updated using a preset equation to obtain a smoothed gland three-dimensional matrix; the preset equation is:

[0038]

[0039] Where Δt is the time step, c is the diffusion coefficient, V(x,y,z,t) is the voxel value at position (x,y,z) and time t, is the gradient of the voxel value at position (x, y, z) and time t, and V(x, y, z, t+Δt) is the voxel value at position (x, y, z) and the next time t+Δt.

[0040] In a second aspect, an embodiment of the present invention provides a three-dimensional reconstruction device, which is applied to an electronic device of an ultrasonic inspection system, wherein the ultrasonic inspection system further includes an ultrasonic probe, and the device includes:

[0041] An information acquisition unit, used for acquiring all ultrasonic images of the target gland and calculating the first coordinate information of each of the ultrasonic images in the world coordinate system; the ultrasonic images are acquired by detection by the ultrasonic probe;

[0042] A mask map acquisition unit, used for inputting each of the ultrasound images into a gland segmentation network to obtain a gland contour mask map, inputting each of the ultrasound images into a nodule detection network to obtain a nodule contour mask map, and mapping each of the nodule contour mask maps to each of the gland contour mask maps to obtain a gland mask map corresponding to each of the ultrasound images;

[0043] a relative pixel distance calculation unit, configured to select any one of the gland mask images as a first gland mask image, select the remaining gland mask images as second gland mask images, and calculate a relative pixel distance of each of the second gland mask images relative to the first gland mask image according to the first coordinate information;

[0044] The three-dimensional matrix creation unit is used to create a three-dimensional matrix, and put the pixel coordinate data of each gland mask image into the three-dimensional matrix according to each relative pixel distance to obtain the gland three-dimensional matrix.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0046] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements any of the above-described methods when executed by a processor.

[0047] The embodiment of the present invention provides a three-dimensional reconstruction method, device, electronic device and storage medium, which obtains all ultrasound images of the target gland, calculates the first coordinate information of each ultrasound image in the world coordinate system, and segments each of the ultrasound images using the gland segmentation network and the nodule detection network to obtain the gland mask images corresponding to the ultrasound images, and calculates the relative pixel distances of all gland mask images relative to a selected gland mask image, and puts the pixel coordinate data of the gland mask image into a three-dimensional matrix according to the relative pixel distance to obtain the gland three-dimensional matrix. Thus, a three-dimensional model of the gland can be quickly constructed. Different nodules can also be marked with different colors to facilitate doctor observation, and the three-dimensional model of the gland can be made closer to reality through smoothing.

[0048] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A schematic structural block diagram of an ultrasonic inspection system provided by an embodiment of the present invention;

[0051] Figure 2 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention;

[0053] Figure 4 Another schematic structural block diagram of an ultrasonic inspection system provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the structure of an ultrasonic probe provided by an embodiment of the invention;

[0055] Figure 6 Another schematic diagram of a three-dimensional reconstruction method provided by an embodiment of the present invention;

[0056] Figure 7 Another schematic diagram of a three-dimensional reconstruction method provided by an embodiment of the present invention;

[0057] Figure 8A schematic structural block diagram of a three-dimensional reconstruction device provided by an embodiment of the present invention.

[0058] Icons: 10 - ultrasonic inspection system; 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - bus; 200 - ultrasonic probe; 201 - vertex; 300 - sensor; 400 - three-dimensional reconstruction device; 410 - information acquisition unit; 420 - mask map acquisition unit; 430 - relative pixel distance calculation unit; 440 - three-dimensional matrix creation unit. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0062] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0063] Thyroid disease is a common endocrine disorder, including hyperthyroidism, hypothyroidism, thyroid nodules, thyroid cancer and many other diseases. Therefore, it is very important to prevent, detect and treat thyroid diseases early. Thyroid ultrasound is an important means of diagnosing thyroid diseases. Through ultrasound examination, doctors can observe the shape, size, structure, texture and presence of nodules of the thyroid gland.

[0064] In medical imaging, existing 3D reconstruction is usually based on data obtained from CT or MRI scans. Through 3D reconstruction, the location of thyroid nodules or tumors can be located, their size and relationship with surrounding structures can be understood, and a more accurate plan can be provided for subsequent treatment.

[0065] The existing thyroid three-dimensional reconstruction requires patients to undergo CT or MRI examinations, and the entire thyroid gland is reconstructed through the data obtained from the tomography. However, for patients, undergoing additional CT or MRI examinations requires more time and money, and also has a certain impact on the patient's body. However, there are also problems with direct three-dimensional reconstruction through thyroid ultrasound images. When scanning the thyroid gland, the entire thyroid gland cannot be scanned in one section. The left lobe, isthmus, and right lobe of the thyroid gland need to be scanned separately. Therefore, direct three-dimensional reconstruction will affect the reconstruction effect.

[0066] Based on the above situation, an embodiment of the present invention provides a three-dimensional reconstruction method, device, electronic device and storage medium, which obtains all ultrasound images of the target gland, calculates the first coordinate information of each ultrasound image in the world coordinate system, and uses the gland segmentation network and the nodule detection network to obtain the gland mask map of the ultrasound image, and puts the gland mask map into the created three-dimensional matrix to obtain the gland three-dimensional matrix, and performs three-dimensional reconstruction of the target gland.

[0067] To explain the embodiments of the present invention in detail, see Figure 1 , Figure 1 A schematic structural block diagram of an ultrasound examination system provided in an embodiment of the present invention, the ultrasound examination system 10 includes an electronic device 100 and an ultrasound probe 200, the ultrasound probe 200 is used to scan the gland to obtain an ultrasound image, and the electronic device 100 is used to process the ultrasound image.

[0068] To implement the process steps and functions of each example of the present invention, please refer to Figure 2 , Figure 2A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0069] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0070] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0071] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0072] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0073] Understandably, Figure 2 The structure shown is for illustration only. The electronic device 100 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0074] The following is an exemplary description of the three-dimensional reconstruction method provided by the present invention. Figure 3 , Figure 3 A schematic flow chart of a three-dimensional reconstruction method provided by an embodiment of the present invention is provided. The method is applied to an electronic device 100 in an ultrasonic inspection system 10. The method includes the following steps:

[0075] S210: Acquire all ultrasound images of the target gland and calculate the first coordinate information of each ultrasound image in the world coordinate system. The ultrasound images are acquired by the ultrasound probe 200.

[0076] S220: Input each ultrasound image into the gland segmentation network to obtain a gland contour mask map, input each ultrasound image into the nodule detection network to obtain a nodule contour mask map, and map each nodule contour mask map to each gland contour mask map to obtain a gland mask map corresponding to each ultrasound image.

[0077] S230: selecting any one gland mask image as the first gland mask image, and taking the remaining gland mask images as the second gland mask images, and calculating the relative pixel distance of each second gland mask image relative to the first gland mask image according to the first coordinate information.

[0078] S240: Create a three-dimensional matrix, and put the pixel coordinate data of each gland mask image into the three-dimensional matrix according to each relative pixel distance to obtain the gland three-dimensional matrix.

[0079] When the doctor scans the target gland with the ultrasound probe 200, the ultrasound image scanned by the ultrasound probe 200 can be stored in the memory 101 in the electronic device 100. The processor 103 can obtain the ultrasound image from the memory 101 and calculate the first coordinate information of the ultrasound image in the world coordinate system based on the information of the ultrasound image. The world coordinate system is pre-set and will not change.

[0080] In order to facilitate three-dimensional reconstruction, the ultrasound image is segmented and recognized using the trained gland segmentation network and nodule detection network. The ultrasound image is input into the gland segmentation network, which will output a two-dimensional mask map of the same size as the ultrasound image, in which the gland area is marked with a specific pixel value, which can be 1-255, and the non-glandular area can be marked as 0 or background value. The ultrasound image is input into the nodule detection network, and the nodule detection network can also output a mask map, in which the nodule area is marked with a specific pixel value, and the background is 0 or another value. Each ultrasound image is input into the gland segmentation network to obtain a gland contour mask map, each ultrasound image is input into the nodule detection network to obtain a nodule contour mask map, and each nodule contour mask map is mapped to each gland contour mask map, so that the gland mask map corresponding to each ultrasound image can be obtained. The gland mask map will be used for subsequent three-dimensional reconstruction.

[0081] Any gland mask image is selected as the first gland mask image. Preferably, the first gland mask image is selected as the first gland mask image, and the remaining gland mask images are used as the second gland mask image. The pixel distance of the second gland mask image relative to the first gland mask image is calculated to provide a basis for filling the gland mask image into a three-dimensional matrix.

[0082] Create an empty three-dimensional matrix, put the pixel coordinate data of each gland mask map into the three-dimensional matrix, and obtain the gland three-dimensional matrix. The gland three-dimensional matrix can be saved as a Neuroimaging Informatics Technology Initiative (NIFTI) file, and can be read using the ITK-based Tool for Multimodal Image Registration and Analysis (ITK-SNAP) tool to view the three-dimensional reconstruction model of the gland containing the nodule location.

[0083] The method obtains the ultrasonic image of the target gland and converts the ultrasonic image into a gland mask map, selects a gland mask map, calculates the relative pixel distances of all other gland mask maps relative to the selected gland mask map, and puts the coordinate data of all gland mask maps into a three-dimensional matrix according to the pixel distance to obtain the gland three-dimensional matrix, so as to quickly construct a three-dimensional model of the gland.

[0084] Optionally, in order to obtain the coordinate information of the ultrasound image in the world coordinate system without affecting the ultrasound detection of the ultrasound probe, see Figure 4 , Figure 5 , Figure 4 Another schematic structural block diagram of an ultrasonic inspection system provided by an embodiment of the present invention, Figure 5 The ultrasonic inspection system 10 may further include a sensor 300, which is disposed on the ultrasonic probe 200 and is offset in one direction by a preset distance relative to the vertex 201 of the ultrasonic probe 200. Figure 6 , Figure 6 Another flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention is provided. Calculating the first coordinate information of each ultrasound image in the world coordinate system may include the following steps:

[0085] S211: construct a local coordinate system with the center of the sensor as the origin, and determine second coordinate information of the vertex of the ultrasound probe in the local coordinate system according to the position of the sensor center and the vertex of the ultrasound probe.

[0086] S212: Based on the first coordinate information calculation formula, first coordinate information of the ultrasound image in the world coordinate system is calculated according to the third coordinate information and the second coordinate information of the sensor in the world coordinate system.

[0087] A local coordinate system with the center of the sensor 300 as the origin is constructed. Since the vertex 201 of the ultrasonic probe and the sensor 300 are pre-set, the second coordinate information of the vertex 201 of the ultrasonic probe in the local coordinate system can be calculated.

[0088] The sensor 300 can obtain its own coordinates, Euler angles, and the rotation matrix corresponding to the Euler angles in the world coordinate system in real time. Based on the first coordinate information calculation formula, the first coordinate information of the vertex 201 of the ultrasound probe in the world coordinate system can be calculated, that is, the first coordinate information of each ultrasound image in the world coordinate system, that is, the first coordinate information of each gland mask image in the world coordinate system. The first coordinate information calculation formula is:

[0089] YPR*(a,b,c)+(x,y,z)

[0090] Among them, (a, b, c) is the second coordinate information, (x, y, z) is the coordinate of the third coordinate information, and Y, P, R is the rotation matrix corresponding to the Euler angle of the third coordinate information.

[0091] There are many ways to map the nodule contour mask map to the gland contour mask map. Direct mapping can be performed to copy each nodule area in the nodule contour mask map directly to the gland contour mask map; or superposition processing can be performed, which involves mixing the pixels of the nodule contour mask map with the corresponding pixels on the gland mask map according to a predetermined rule or weight. In order to retain the relative intensity information of the nodule, the steps of mapping each nodule contour mask map to each gland contour mask map to obtain the gland mask map corresponding to each ultrasound image may include:

[0092] When the nodule contour mask image is not empty, the superposition value of each pixel in the nodule contour mask image is calculated according to the preset weight value.

[0093] The calculated superposition values ​​are added to the corresponding pixels of the gland contour mask map to obtain the gland mask map.

[0094] First, determine whether the nodule mask image of the target gland has nodule data. If there is no nodule data, no overlay processing is required.

[0095] If there is nodule data, the superposition value of each pixel in the nodule contour mask is calculated according to the preset weight value. The weight value is used to control the superposition intensity of the nodule information in the gland mask map. For each pixel in the nodule mask map, the superposition effect in the gland mask map is calculated according to the weight value. For example, if the pixel value of the nodule contour mask map is 1 and the pixel value of the gland contour mask map is 0.5, the superimposed value is 0.5+(1*weight value). The gland mask map can be obtained by adding each superposition value to the corresponding pixel of the gland contour mask map.

[0096] In order to facilitate the distinction of different nodules in the target gland, the nodule data in the gland mask image can be further processed, such as color marking. The three-dimensional reconstruction method can also include:

[0097] Detect whether there is nodule data in the gland mask map; wherein a gland mask map includes at least zero nodule data.

[0098] If yes, based on the gland mask map, each nodule data is marked with a different color according to each nodule data.

[0099] Check whether there is nodule data in the gland mask image. If not, no color marking is required.

[0100] If there is nodule data, there are often multiple nodules in the gland. First identify the location of each nodule, and then mark each nodule. There are many ways to mark, which can be changing pixel values, color marking, multi-channel marking, etc. Among them, changing pixel values ​​is to change the pixel value of the nodule area to a specific marking value on the gland mask map; color marking is to assign different colors to each nodule; multi-channel marking is in a multi-channel image, a separate channel can be used to represent the nodule information.

[0101] In order to conveniently fill the pixel coordinate data of the gland mask image into the three-dimensional matrix, the pixel coordinates of each gland mask image can be calculated. Preferably, a gland mask image is selected as a reference for calculation. Figure 7 , Figure 7 Another flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention is shown in step S230, wherein the following steps may be performed:

[0102] S231: Subtract the first coordinate information of the first ultrasonic image from the first coordinate information of each second ultrasonic image to obtain the relative physical coordinates of each second ultrasonic image.

[0103] S232: Convert each relative physical coordinate into a relative pixel distance according to the pixel distance conversion coefficient of the ultrasound image.

[0104] According to the first coordinate information of each ultrasound image in step S210, it is recorded as (x i ,y i , z i ), the first coordinate information of the first ultrasonic image is recorded as (x1, y1, z1), and the first coordinate information of the second ultrasonic image is recorded as (x2, y2, z2), then the relative physical coordinates of the second ultrasonic image are (x2-x1, y2-y1, z2-z1), recorded as (Δx, Δy, Δz).

[0105] The pixel distance conversion coefficient of the ultrasound image is recorded as a, and the relative pixel distance is (Δx, Δy, Δz)*a, and rounding is performed because Δz is calculated from the coordinates measured by the sensor 300 and the Euler angle, which may not be an integer, while the pixel needs to be an integer. The pixel distance conversion coefficient a can be calculated based on the physical size and pixel size of the ultrasound image. For example, if the imaging depth is 50 mm and the image width is 512 pixels, the conversion coefficient is

[0106] There are many ways to create a three-dimensional matrix. For example, a larger three-dimensional matrix can be directly created to accommodate all gland mask images without wasting resources. In step S240, the step of creating a three-dimensional matrix may include:

[0107] The dimension of the three-dimensional matrix is ​​determined according to the pixels of the ultrasound image and the relative pixel distance.

[0108] Initialize a three-dimensional matrix according to the dimension.

[0109] Specifically, the pixel size of the ultrasound image plus the longest relative pixel distance Δmx, Δmy are used as the width and height of the three-dimensional matrix, and the longest Δmz is used as the depth. For example, if the pixel size of the ultrasound image is (w, h), then the dimension of the three-dimensional matrix is ​​(w+Δmx, h+Δmy, Δmz). According to this dimension, the three-dimensional matrix can be initialized.

[0110] After having the mask images of each gland, the relative pixel distances and the three-dimensional matrix, it is only necessary to fill the mask images of each gland into the three-dimensional matrix to obtain the three-dimensional matrix of the gland. Specifically, according to the relative pixel distances, the pixel coordinate data of each gland mask image is placed into the three-dimensional matrix, and the step of obtaining the three-dimensional matrix of the gland may include:

[0111] According to the pixel coordinates of the first gland mask map, the first gland mask map is placed into a three-dimensional matrix.

[0112] According to the pixel coordinates and relative pixel distances of the second gland mask image, the second gland mask image is placed into a three-dimensional matrix to obtain a gland three-dimensional matrix.

[0113] The pixel coordinates of the first gland mask image are directly placed into the three-dimensional matrix. For example, the pixel coordinates of the first gland mask image are (x1, y1). When the pixel coordinates of the first gland mask image are placed at a position where the depth of the three-dimensional matrix is ​​0, the position in the three-dimensional matrix is ​​(x1, y1, 0).

[0114] The remaining second gland mask images are placed in a three-dimensional matrix according to their relative pixel distances with the first gland mask image. For example, if the pixel coordinates on the second gland mask image are (x2, y2), and the relative pixel distance of the second gland mask image relative to the first gland mask image is (Δx, Δy, Δz)*a, then the new pixel coordinates are (x2+Δx*a, y2+Δy*a, Δz*a).

[0115] In order to make the three-dimensional model of the gland smoother and closer to reality, the three-dimensional matrix can also be smoothed. The smoothing method can include: Gaussian blur, surface smoothing, bilateral filtering, anisotropic diffusion, etc. In order to keep the gland boundary clear during smoothing, this embodiment uses anisotropic diffusion for processing. Specifically, the three-dimensional reconstruction method can also include:

[0116] The gradient and gradient amplitude of the glandular three-dimensional matrix were calculated.

[0117] According to the preset diffusion function, the diffusion coefficient of each voxel in the gland three-dimensional matrix is ​​calculated; the diffusion coefficient is:

[0118]

[0119] Where c is the diffusion coefficient, g is the diffusion function, is the gradient amplitude, and k is a positive constant.

[0120] The constant k is set according to actual needs to control the diffusion rate and the sharpness of the edge.

[0121] Each voxel is updated using a preset equation to obtain a smoothed three-dimensional matrix of the gland; the preset equation is:

[0122]

[0123] Where Δt is the time step, c is the diffusion coefficient, V(x,y,z,t) is the voxel value at position (x,y,z) and time t, is the gradient of the voxel value at position (x, y, z) and time t, and V(x, y, z, t+Δt) is the voxel value at position (x, y, z) and the next time t+Δt.

[0124] Let the three-dimensional matrix be V, select an initial time t=0 and a maximum number of iterations N, and set the parameter k of the diffusion function g.

[0125] Compute the gradient of each voxel in V and the gradient amplitude Using the selected diffusion function g, the gradient magnitude of each voxel is Calculate the diffusion coefficient c, and for each voxel, update its value according to the preset equation. Repeat these steps until the maximum number of iterations N is reached or until the three-dimensional matrix V converges to a stable state. Complete the smoothing of the three-dimensional matrix.

[0126] Furthermore, the present invention also provides a three-dimensional reconstruction device 400, which is applied to the electronic device 100 of the ultrasonic inspection system 10. Figure 8 , Figure 8 A schematic structural block diagram of a three-dimensional reconstruction device provided in an embodiment of the present invention, the three-dimensional reconstruction device 400 includes: an information acquisition unit 410, a mask map acquisition unit 420, a relative pixel distance calculation unit 430 and a three-dimensional matrix creation unit 440:

[0127] The information acquisition unit 410 is used to acquire all ultrasonic images of the target gland and calculate the first coordinate information of each ultrasonic image in the world coordinate system; wherein the ultrasonic image is acquired by detection by an ultrasonic probe.

[0128] The mask map acquisition unit 420 is used to input each ultrasound image into the gland segmentation network to obtain a gland contour mask map, input each ultrasound image into the nodule detection network to obtain a nodule contour mask map, and map each nodule contour mask map to each gland contour mask map to obtain a gland mask map corresponding to each ultrasound image.

[0129] The relative pixel distance calculation unit 430 is used to select any gland mask image as the first gland mask image, use the remaining gland mask images as the second gland mask images, and calculate the relative pixel distance of each second gland mask image relative to the first gland mask image according to the first coordinate information.

[0130] The three-dimensional matrix creation unit 440 is used to create a three-dimensional matrix, and put the pixel coordinate data of each gland mask image into the three-dimensional matrix according to each relative pixel distance to obtain the gland three-dimensional matrix.

[0131] In summary, the embodiments of the present invention provide a three-dimensional reconstruction method, device, electronic device and storage medium, which obtain an ultrasonic image of a target gland and convert the ultrasonic image into a gland mask image, mark each nodule in the gland mask image, select a gland mask image, calculate the relative pixel distances of all other gland mask images relative to the selected gland mask image, put the coordinate data of all gland mask images into a three-dimensional matrix according to the pixel distance, obtain the gland three-dimensional matrix, and smooth the gland three-dimensional matrix, so as to quickly construct a three-dimensional model of the gland, distinguish different nodules, and make the three-dimensional model smoother and more realistic.

[0132] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0134] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0135] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0136] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A three-dimensional reconstruction method, characterized in that: An electronic device applied to an ultrasonic inspection system, wherein the ultrasonic inspection system further comprises an ultrasonic probe, and the method comprises: Acquire all ultrasonic images of the target gland, and calculate the first coordinate information of each ultrasonic image in the world coordinate system; the ultrasonic image is detected and obtained by the ultrasonic probe; Inputting each of the ultrasound images into a gland segmentation network to obtain a gland contour mask map, inputting each of the ultrasound images into a nodule detection network to obtain a nodule contour mask map, and mapping each of the nodule contour mask maps to each of the gland contour mask maps to obtain a gland mask map corresponding to each of the ultrasound images; Selecting any one of the gland mask images as a first gland mask image, and taking the remaining gland mask images as second gland mask images, and calculating a relative pixel distance of each of the second gland mask images relative to the first gland mask image according to the first coordinate information; A three-dimensional matrix is ​​created, and pixel coordinate data of each gland mask image is placed into the three-dimensional matrix according to each relative pixel distance, so as to obtain a gland three-dimensional matrix.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The ultrasonic inspection system further includes a sensor, which is disposed on the ultrasonic probe and is offset in one direction by a preset distance relative to the vertex of the ultrasonic probe. The step of calculating the first coordinate information of each of the ultrasonic images in the world coordinate system includes: Constructing a local coordinate system with the center of the sensor as the origin, and determining second coordinate information of the vertex in the local coordinate system according to the positions of the center and the vertex; Based on the first coordinate information calculation formula, the first coordinate information of the ultrasonic image in the world coordinate system is calculated according to the third coordinate information of the sensor in the world coordinate system and the second coordinate information.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The calculation formula of the first coordinate information is: YPR*(a,b,c)+(x,y,z) Among them, (a, b, c) is the second coordinate information, (x, y, z) is the coordinate of the third coordinate information, and Y, P, R is the rotation matrix corresponding to the Euler angle of the third coordinate information.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of mapping each of the nodule contour mask images to each of the gland contour mask images to obtain a gland mask image corresponding to each of the ultrasound images comprises: When the nodule contour mask image is not empty, calculating the superposition value of each pixel in the nodule contour mask image according to a preset weight value; The calculated superposition values ​​are added to corresponding pixels of the gland contour mask image to obtain the gland mask image.

5. The three-dimensional reconstruction method according to claim 4, characterized in that: The method further comprises: Detecting whether there is nodule data in the gland mask image; a gland mask image includes at least zero nodule data; If yes, based on the gland mask image, each nodule data is marked with a different color according to each nodule data.

6. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of selecting any one of the gland mask images as the first gland mask image, using the remaining gland mask images as the second gland mask images, and calculating the relative pixel distance of each of the second gland mask images relative to the first gland mask image according to the first coordinate information comprises: subtracting the first coordinate information of the first gland mask image from the first coordinate information of each of the second gland mask images to obtain the relative physical coordinates of each of the second gland mask images; Each of the relative physical coordinates is converted into a relative pixel distance according to a pixel distance conversion coefficient of the ultrasonic image.

7. The three-dimensional reconstruction method according to claim 6, characterized in that: The step of creating a three-dimensional matrix includes: Determining the dimension of the three-dimensional matrix according to the pixels of the ultrasound image and the relative pixel distance; Initialize the three-dimensional matrix according to the dimension.

8. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of placing the pixel coordinate data of each gland mask image into the three-dimensional matrix according to each relative pixel distance to obtain the gland three-dimensional matrix comprises: According to pixel coordinates of the first gland mask image, placing the first gland mask image into the three-dimensional matrix; According to the pixel coordinates of the second gland mask image and the relative pixel distance, the second gland mask image is placed in the three-dimensional matrix to obtain the gland three-dimensional matrix.

9. The three-dimensional reconstruction method according to claim 1, characterized in that: The method further comprises: Calculating the gradient and gradient amplitude of the three-dimensional matrix of the gland; According to the preset diffusion function, the diffusion coefficient of each voxel of the gland three-dimensional matrix is ​​calculated; the diffusion coefficient is: Wherein, c is the diffusion coefficient, g is the diffusion function, is the gradient amplitude, k is a positive constant; Each voxel is updated using a preset equation to obtain a smoothed gland three-dimensional matrix; the preset equation is: Where Δt is the time step, c is the diffusion coefficient, V(x,y,z,t) is the voxel value at position (x,y,z) and time t, is the gradient of the voxel value at position (x, y, z) and time t, and V(x, y, z, t+Δt) is the voxel value at position (x, y, z) and the next time t+Δt.

10. A three-dimensional reconstruction device, characterized in that: An electronic device applied to an ultrasonic inspection system, wherein the ultrasonic inspection system further comprises an ultrasonic probe, and the device comprises: An information acquisition unit, used for acquiring all ultrasonic images of the target gland and calculating the first coordinate information of each of the ultrasonic images in the world coordinate system; the ultrasonic images are acquired by detection by the ultrasonic probe; A mask map acquisition unit, used for inputting each of the ultrasound images into a gland segmentation network to obtain a gland contour mask map, inputting each of the ultrasound images into a nodule detection network to obtain a nodule contour mask map, and mapping each of the nodule contour mask maps to each of the gland contour mask maps to obtain a gland mask map corresponding to each of the ultrasound images; a relative pixel distance calculation unit, configured to select any one of the gland mask images as a first gland mask image, select the remaining gland mask images as second gland mask images, and calculate a relative pixel distance of each of the second gland mask images relative to the first gland mask image according to the first coordinate information; The three-dimensional matrix creation unit is used to create a three-dimensional matrix, and put the pixel coordinate data of each gland mask image into the three-dimensional matrix according to each relative pixel distance to obtain the gland three-dimensional matrix.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images